Image processing method, system, computing device and storage medium
By using complete model training in multimodal image processing, the location difference data and image features are obtained, processing efficiency and stability problems are solved, and faster and more accurate target object segmentation is achieved.
Patent Information
- Application Number
- CN201910852977.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-09-10
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2039-09-10
AI Technical Summary
The prior art has poor processing efficiency and stability in multimodal image processing, making it difficult to quickly and accurately segment the target object.
By acquiring two different images for the same target object, using the model to be trained to obtain position difference data and image features, a complete model is trained to perform segmentation processing of the target object, reducing the dependence and processing time of multiple models.
It improves the speed and stability of image segmentation processing, enhances the integrity and accuracy of processing, and reduces dependence on other factors.
Smart Images

Figure CN112561914B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to an image processing method, system, computing device and storage medium. Background Art
[0002] With the development of information technology, computer vision has been well developed. Computer vision enables machines to "see" things, even things that humans cannot see. Especially for image processing, before image processing, people are often interested in a certain part of the image, so the part of interest needs to be processed. However, for multiple types of images, the processing efficiency and stability are poor when processing images. Summary of the invention
[0003] Various aspects of the present application provide an image processing method, system, computing device, and storage medium to improve the processing efficiency and processing stability of segmenting target objects from multi-modal images.
[0004] An embodiment of the present application provides an image processing method, including: acquiring a first image and a second image, the first image serving as a reference image, the second image serving as an image to be calibrated, the first image and the second image being different images for the same target object; inputting the first image and the second image into a model to be trained, acquiring position difference data between the first image and the second image and the target object, the target object acquired having image features of the first image and the second image; and training the model to be trained according to the target object and the position difference data.
[0005] An embodiment of the present application also provides an image processing method, including: acquiring a first image and a second image, the first image serving as a reference image, the second image serving as an image to be calibrated, the first image and the second image being different images for the same target object; inputting the first image and the second image into an image processing model to acquire the target object, the acquired target object having image features of the first image and the second image.
[0006] An embodiment of the present application also provides an image processing system, including: a first device and a second device; the first device acquires a first image and a second image, the first image serves as a reference image, the second image serves as an image to be calibrated, and the first image and the second image are different images for the same target object; the first image and the second image are sent to the second device; the second device inputs the first image and the second image into an image processing model to acquire the target object, the acquired target object has image features of the first image and the second image; and the target object is sent to the first device.
[0007] An embodiment of the present application also provides an image processing method, including: acquiring a first image and a second image, the first image serving as a reference image, the second image serving as an image to be calibrated, the first image and the second image being different images for the same target object; inputting the first image and the second image into a model to be trained, acquiring position difference data between the first image and the second image through a first sub-model in the model to be trained, acquiring the target object through the position difference data and the second sub-model in the model to be trained, the acquired target object having image features of the first image and the second image, and the training parameters of the first sub-model and the second sub-model being the same; training the training parameters according to the target object and the position difference data, thereby training the model to be trained.
[0008] An embodiment of the present application also provides an image processing method, including: acquiring a first medical image and a second medical image, wherein the first medical image and the second medical image are different modality images for the same abnormal area; inputting the first medical image and the second medical image into an image processing model to acquire an abnormal area in the spliced first medical image and the second medical image, wherein the acquired abnormal area has image features of the first medical image and the second medical image; and providing an image of the abnormal area, wherein the first medical image is used as a reference image and the second medical image is used as an image to be calibrated.
[0009] An embodiment of the present application also provides an image processing method, including: acquiring a first medical image and a second medical image, wherein the first medical image and the second medical image are different modality images for the same abnormal area; inputting the first medical image and the second medical image into an image processing model, acquiring the abnormal area in the spliced first medical image and the second medical image, and determining the location of the abnormal area, wherein the acquired abnormal area has image features of the first medical image and the second medical image; providing the image and location of the abnormal area, wherein the first medical image is used as a reference image, and the second medical image is used as an image to be calibrated.
[0010] An embodiment of the present application also provides an image processing method, including: acquiring a first monitoring image and a second monitoring image, the first monitoring image and the second monitoring image are from different monitoring devices, the first monitoring image is used as a reference image, the second monitoring image is used as an image to be calibrated, and the first monitoring image and the second monitoring image are different images for the same monitoring target object; inputting the first monitoring image and the second monitoring image into an image processing model to acquire the monitoring target object, the acquired monitoring target object has image features of the first monitoring image and the second monitoring image; and identifying information to be found in the monitoring target object.
[0011] An embodiment of the present application also provides a video processing method, including: obtaining a first surveillance video and a second surveillance video, the first surveillance video and the second surveillance video coming from different surveillance devices; obtaining a first surveillance image from the first surveillance video, and obtaining a second surveillance image corresponding to the first surveillance video from the second surveillance video, the first surveillance image serving as a reference image, the second surveillance image serving as an image to be calibrated, the first surveillance image and the second surveillance image being different images for the same surveillance target object; inputting the first surveillance image and the second surveillance image into an image processing model to obtain the surveillance target object, the acquired surveillance target object having image features of the first surveillance image and the second surveillance image; and identifying information to be found in the surveillance target object.
[0012] An embodiment of the present application also provides a method for obtaining an image processing, including: a first medical image and a second medical image, wherein the first medical image and the second medical image are different modality images for the same abnormal area, the first medical image serves as a reference image, and the second medical image serves as an image to be calibrated; the first medical image and the second medical image are input into a model to be trained, and position difference data between the first medical image and the second medical image and the abnormal area are obtained, wherein the abnormal area obtained has image features of the first image and the second image; and the model to be trained is trained according to the abnormal area and the position difference data.
[0013] An embodiment of the present application also provides an image processing method, including: acquiring a first medical image and a second medical image, wherein the first medical image and the second medical image are different modality images for the same abnormal area; inputting the first medical image and the second medical image into an image processing model, acquiring the abnormal area in the spliced first medical image and the second medical image, wherein the acquired abnormal area has image features of the first medical image and the second medical image; acquiring model data of a digital virtual person; and fusing the model data with the abnormal area so that the fused digital virtual person has the abnormal area.
[0014] An embodiment of the present application also provides a computing device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program, so as to: obtain a first image and a second image, the first image serving as a reference image, the second image serving as an image to be calibrated, the first image and the second image being different images for the same target object; inputting the first image and the second image into a model to be trained, obtaining position difference data between the first image and the second image and the target object, the target object obtained having image features of the first image and the second image; and training the model to be trained according to the target object and the position difference data.
[0015] An embodiment of the present application also provides a computing device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to: acquire a first image and a second image, the first image serving as a reference image, the second image serving as an image to be calibrated, the first image and the second image acquired being different images for the same target object; input the first image and the second image into an image processing model to acquire the target object, the target object having image features of the first image and the second image.
[0016] An embodiment of the present application also provides a computing device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program, so as to: obtain a first monitoring image and a second monitoring image, the first monitoring image and the second monitoring image are from different monitoring devices, the first monitoring image is used as a reference image, the second monitoring image is used as an image to be calibrated, and the first monitoring image and the second monitoring image are different images for the same monitoring target object; input the first monitoring image and the second monitoring image into an image processing model to obtain the monitoring target object, the obtained monitoring target object has the image features of the first monitoring image and the second monitoring image; and identify the information to be found in the monitoring target object.
[0017] An embodiment of the present application also provides a computing device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program, so as to: obtain a first surveillance video and a second surveillance video, the first surveillance video and the second surveillance video are from different surveillance devices; obtain a first surveillance image from the first surveillance video, and obtain a second surveillance image corresponding to the first surveillance video from the second surveillance video, the first surveillance image is used as a reference image, the second surveillance image is used as an image to be calibrated, and the first surveillance image and the second surveillance image are different images for the same surveillance target object; input the first surveillance image and the second surveillance image into an image processing model to obtain the surveillance target object, the acquired surveillance target object has the image features of the first surveillance image and the second surveillance image; identify the information to be found in the surveillance target object.
[0018] An embodiment of the present application also provides a computing device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program, so as to: obtain a first image and a second image, the first image is used as a reference image, and the second image is used as an image to be calibrated, and the first image and the second image are different images for the same target object; input the first image and the second image into a model to be trained, and obtain position difference data between the first image and the second image through a first sub-model in the model to be trained, and obtain the target object through the position difference data and the second sub-model in the model to be trained, the obtained target object has the image features of the first image and the second image, and the training parameters of the first sub-model and the second sub-model are the same; train the training parameters according to the target object and the position difference data, so as to train the model to be trained.
[0019] An embodiment of the present application also provides a computing device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program, so as to: obtain a first medical image and a second medical image, wherein the first medical image and the second medical image are different modality images for the same abnormal area; input the first medical image and the second medical image into an image processing model to obtain an abnormal area in the spliced first medical image and the second medical image, wherein the acquired abnormal area has image features of the first medical image and the second medical image; and provide an image of the abnormal area, wherein the first medical image is used as a reference image and the second medical image is used as an image to be calibrated.
[0020] An embodiment of the present application also provides a computing device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program, so as to: obtain a first medical image and a second medical image, wherein the first medical image and the second medical image are different modality images for the same abnormal area; input the first medical image and the second medical image into an image processing model, obtain the abnormal area in the spliced first medical image and the second medical image, and determine the location of the abnormal area, wherein the acquired abnormal area has the image features of the first medical image and the second medical image; provide the image and location of the abnormal area, wherein the first medical image is used as a reference image and the second medical image is used as an image to be calibrated.
[0021] An embodiment of the present application also provides a computing device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program, so as to: obtain a first medical image and a second medical image, the first medical image and the second medical image are different modality images for the same abnormal area, the first medical image is used as a reference image, and the second medical image is used as an image to be calibrated; the first medical image and the second medical image are input into a model to be trained, and position difference data between the first medical image and the second medical image and the abnormal area are obtained, and the abnormal area obtained has image features of the first image and the second image; and the model to be trained is trained according to the abnormal area and the position difference data.
[0022] An embodiment of the present application also provides a computing device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program, so as to: obtain a first medical image and a second medical image, wherein the first medical image and the second medical image are different modality images for the same abnormal area; input the first medical image and the second medical image into an image processing model to obtain the abnormal area in the spliced first medical image and the second medical image, wherein the acquired abnormal area has the image features of the first medical image and the second medical image; obtain model data of a digital virtual person; and fuse the model data with the abnormal area so that the fused digital virtual person has the abnormal area.
[0023] An embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by one or more processors, the one or more processors implement the steps in the above method.
[0024] In an embodiment of the present application, the first image and the second image are input into the model to be trained, and the position difference data of the first image and the second image and the target object are obtained. The obtained target object has the image features of the first image and the second image; the first image is used as a reference image, and the second image is used as an image to be calibrated. According to the target object and the position difference data, the model to be trained is trained, so that a complete model after training can perform target object segmentation processing on multimodal images. Compared with obtaining position difference data and target objects for multimodal images respectively through different models, thereby realizing target object segmentation processing on multimodal images through multiple models, the total time for processing and obtaining position difference data and target objects is reduced, and the total speed of image segmentation processing is improved. And because the segmentation processing of multimodal images is realized by training a complete model, compared with realizing the segmentation processing of multimodal images by multiple models, the complete model after training can improve the stability and accuracy of image processing, reduce the dependence on other factors in the segmentation processing of multimodal images, and enhance the integrity of image processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0026] Figure 1A A structural schematic diagram of an image processing system according to an exemplary embodiment of the present application;
[0027] Figure 1BA structural schematic diagram of an image processing system according to an exemplary embodiment of the present application;
[0028] Figure 2 A schematic flow chart of an image processing method according to an exemplary embodiment of the present application;
[0029] Figure 3 A schematic diagram of model training provided for an exemplary embodiment of the present application;
[0030] Figure 4A A schematic flow chart of an image processing method according to an exemplary embodiment of the present application;
[0031] Figure 4B A schematic diagram of image processing according to an exemplary embodiment of the present application;
[0032] Figure 5A A schematic flow chart of an image processing method according to an exemplary embodiment of the present application;
[0033] Figure 5B A schematic diagram of model training according to an exemplary embodiment of the present application;
[0034] Fig. 6A A schematic flow chart of an image processing method according to an exemplary embodiment of the present application;
[0035] Figure 6B A schematic diagram of image processing according to an exemplary embodiment of the present application;
[0036] Figure 7 A schematic flow chart of an image processing method according to an exemplary embodiment of the present application;
[0037] Fig. 8A A schematic flow chart of an image processing method according to an exemplary embodiment of the present application;
[0038] Figure 8B A schematic diagram of image processing according to an exemplary embodiment of the present application;
[0039] Fig. 9 A flowchart of a video processing method according to an exemplary embodiment of the present application;
[0040] Fig.10 A schematic diagram of the structure of an image processing device provided by an exemplary embodiment of the present application;
[0041] Fig.11 A schematic diagram of the structure of a computing device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0043] In image segmentation scenarios, it is often necessary to use multimodal image information to more accurately determine the target object, so it is often necessary to scan multiple images of different modalities. These images often have morphological differences due to some objective factors, such as the movement of the photographed object, differences in scanning equipment, etc., resulting in the inability to quickly perform image segmentation.
[0044] The embodiment of the present application provides an end-to-end solution that improves processing efficiency while maintaining accurate processing accuracy and optimizing the overall solution.
[0045] In an embodiment of the present application, the first image and the second image are input into the model to be trained, and the position difference data of the first image and the second image and the target object are obtained, and the target object has the image features of the first image and the second image; the first image is used as a reference image, and the second image is used as an image to be calibrated, and the model to be trained is trained according to the target object and the position difference data, so that a complete model after training can perform segmentation processing on the multimodal image for the target object, compared with obtaining the position difference data and the target object for the multimodal image respectively through different models, thereby realizing the segmentation processing of the target object for the multimodal image through multiple models, the total time for processing the acquisition of the position difference data and the target object is reduced, and the total speed of the image segmentation processing is improved. And because the segmentation processing of the multimodal image is realized by training a complete model, compared with realizing the segmentation processing of the multimodal image by multiple models, the complete model after training can improve the stability and accuracy of the image processing, reduce the dependence on other factors in the segmentation processing of the multimodal image, and enhance the integrity of the image processing.
[0046] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0047] Figure 1A The following is a schematic diagram of the structure of an image processing system provided by an exemplary embodiment of the present application. Figure 1A As shown, the processing system 100A may include: a first device 101 and a second device 102 .
[0048] Among them, the first device 101 can be a device with certain computing capabilities. The basic structure of the first device 101 may include: at least one processor. The number of processors may depend on the configuration and type of the device with certain computing capabilities. The device with certain computing capabilities may also include a memory, which may be volatile, such as RAM, or non-volatile, such as read-only memory (ROM), flash memory, etc., or may include both types at the same time. The memory usually stores an operating system (OS), one or more application programs, and may also store program data, etc. In addition to the processing unit and the memory, the device with certain computing capabilities also includes some basic configurations, such as a network card chip, an IO bus, a display component, and some peripheral devices. Optionally, some peripheral devices may include, for example, a keyboard, an input pen, etc. Other peripheral devices are well known in the art and will not be described in detail here. Optionally, the first device 101 may be a smart terminal, a vehicle-mounted terminal, a drone, etc.
[0049] The second device 102 refers to a device that can provide computing and processing services in a network virtual environment, and can refer to a device that uses the network to perform model training. In terms of physical implementation, the second device 102 can be any device that can provide computing services, respond to service requests, and perform processing, such as a conventional server, a cloud server, an image tracking device set on the roadside, an image tracking device set indoors, etc. The second device 102 mainly includes a processor, a hard disk, a memory, a system bus, etc., which is similar to a general computer architecture.
[0050] In the example of the present application, the first device 101 acquires a first image and a second image, the first image serves as a reference image, the second image serves as an image to be calibrated, and the first image and the second image are different images of the same target object; and sends the first image and the second image to the second device 102.
[0051] The second device 102 inputs the first image and the second image into the image processing model to obtain a target object, where the obtained target object has image features of the first image and the second image; and sends the target object to the first device 101.
[0052] In some examples, the second device 102 obtains position difference data between the first image and the second image; and obtains the target object according to the difference data.
[0053] In some examples, the second device 102 acquires the target object, including: processing the second image according to the position difference data; stitching the first image and the processed second image to obtain a stitched image; and segmenting the target object from the stitched image.
[0054] In some instances, the second device 102 acquires a first image and a second image, wherein the first image serves as a reference image and the second image serves as an image to be calibrated, and the first image and the second image are different images of the same target object; the first image and the second image are input into a model to be trained, and position difference data between the first image and the second image and the target object are acquired, wherein the acquired target object has image features of the first image and the second image; the model to be trained is trained according to the target object and the position difference data, and an image processing model is generated.
[0055] It should be noted that the specific implementation of the training model executed by the second device 102 can also be executed by the first device 101, and the target object can be directly acquired through the image processing model generated in the first device 101, and the final acquired target object can be displayed through the first device 101.
[0056] In the application scenario of the embodiment of the present application, Figure 1B As shown, for example, a user locally obtains two images of different modalities from a first device 101, such as a computer, such as a CT (Computed Tomography) image and an X-ray image, and sends the two images to a second device 102, such as a server, and specifies the CT image as a reference image. After receiving the two images, the server can process the two images according to an image processing model deployed in the server. The server can determine the position difference data of the X-ray image relative to the CT image through the model, and then process the X-ray image according to the position difference data through the model, and splice the processed X-ray image with the CT image, and segment the target object from the spliced image, such as a muscle tissue image in the spliced image. The server sends the target object to the computer, so that the computer receives the target object and displays the target object, and the user can visually observe the target object.
[0057] In addition, the second device 102, such as a server, can train the image processing model before deploying the image processing model. The training process can be that the server obtains multiple training samples, and the training samples can include two categories, one belonging to the reference image (i.e., the first image) and the other belonging to the image to be calibrated (the second image). At the same time, the server also obtains a class of samples, which are reference images of the target object. Among them, the first image and the second image are corresponding images, such as images that both belong to a certain muscle tissue. The server inputs the first image and the second image into the model to be trained, i.e., the initial model, determines the position difference data of the second image relative to the first image through the model, processes the second image according to the position difference data, splices the processed second image with the first image, and segments the target object in the spliced image through the model. Thereby determining the loss function between the target object and the reference image of the target object, and the loss function between the processed second image and the first image. The initial parameters of the model, i.e., the training parameters, are adjusted according to the sum of these two loss functions. After the adjustment, continue to train the model and adjust the training parameters to iterate the training parameters until the training parameters are adjusted so that the sum of the loss function obtained by the model is within the expected value range, or the loss function is within the expected value range, or the number of iterations reaches the expected number, etc. Generate the final image processing model.
[0058] It should be noted that the first image may be one of multiple images, which may belong to a sample set, and the second image may be one of multiple images, which may belong to a sample set, but the first image and the second image taken out from the sample set correspond to each other and constitute a group of training samples, such as an image of a certain muscle tissue.
[0059] In addition, the first image and the second image may be directly used in advance as a group of training samples, but there are multiple groups of training samples.
[0060] In the above embodiment, the first device 101 can establish a network connection with the second device 102, and the network connection can be a wireless connection. If the first device 101 and the second device 102 are connected in communication, the network standard of the mobile network can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), WiMax, 5G, etc.
[0061] The following describes in detail the model training process and model usage process of the second device 102 in conjunction with the method embodiment.
[0062] Figure 2The flowchart of the image processing method of an exemplary embodiment of the present application is shown in FIG. The method 200 provided in the embodiment of the present application is executed by a second device, such as a server, and the method 200 includes the following steps:
[0063] 201: Acquire a first image and a second image, wherein the first image is used as a reference image and the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object.
[0064] 202: Input the first image and the second image into the model to be trained, obtain position difference data between the first image and the second image and a target object, and the obtained target object has image features of the first image and the second image.
[0065] 203: Train the model to be trained according to the target object and the position difference data.
[0066] It should be noted that, in addition to being executed by a server, the method 200 can also be directly executed by a terminal with powerful intelligent functions, such as a computer in a smart terminal. The device executing the method 200 can be deployed with a display component, such as a display screen, and the obtained final image result can be directly displayed through the display screen for users to view.
[0067] The following is a detailed description of the above steps:
[0068] 201: Acquire a first image and a second image, wherein the first image is used as a reference image and the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object.
[0069] The reference image refers to a benchmark image, that is, the image is used as the base image of the two images, and the other image is to be adjusted or processed based on the image. The reference image can be a medical image, such as a magnetic resonance image, a CT image, an ultrasound image, an X-ray image, etc. It can also be a monitoring image, such as an image taken from different angles or different shooting devices.
[0070] The image to be calibrated refers to an image that needs to be adjusted in position, and the position adjustment is performed according to the reference image. The image to be calibrated can be a medical image, such as a magnetic resonance image, a CT image, an ultrasound image, an X-ray image, etc. It can also be a monitoring image, such as an image taken at different angles or with different shooting equipment.
[0071] It should be noted that the reference image and the image to be calibrated are determined according to service requirements in actual application scenarios.
[0072] According to the above description, it can be known that the image to be calibrated is adjusted according to the reference image, and what is adjusted is the positional relationship between the two, that is, the coordinate relationship.
[0073] It should also be noted that the first image and the second image are a group of training samples. The images in each group of training samples are for a certain target object, such as an object image, a muscle tissue image, etc. The target objects of multiple groups of training samples can be the same or different.
[0074] For example, according to the foregoing description, the server may obtain multiple groups of training samples from its database or other server nodes, and obtain the first image and the second image from each group of training samples.
[0075] It should be noted that, during the model training process, a group of training samples can train the model once, so the specific implementation method of training the model for each group of training samples is the same. The specific implementation method of training the model for a group of training samples is described below.
[0076] 202: Input the first image and the second image into the model to be trained, obtain position difference data between the first image and the second image and a target object, and the obtained target object has image features of the first image and the second image.
[0077] The model to be trained may also be referred to as an initial model, and the training parameters in the model are all initial parameters and have been preset. The training of the model is completed by training the training parameters. For example, the structure of the initial model may be a U-NET network structure or a U-NET++ network structure.
[0078] Position difference data, also called deformation field, refers to: the pixel-to-pixel position relationship (eg, coordinate offset) between two images; such as a set of coordinate offsets of each pixel.
[0079] The target object refers to an object that is ultimately to be acquired from at least two multimodal images, such as a vehicle image, a person image, a muscle tissue image, etc.
[0080] Image features refer to attributes that describe image information, such as image color, texture, spatial position relationship, etc. The acquired target object includes both the image features of the first image and the image features of the second image. It should be understood that the image features here are image features corresponding to the target object in each image.
[0081] For example, according to the above, if Figure 3 As shown, after acquiring a set of first images "CT images" and second images "X-ray images", the server 101 inputs the first images and the second images into a preset initial model. The initial model outputs the vehicle images in the first image and the second image, and acquires the deformation field of the second image relative to the first image through the training parameters in the initial model.
[0082] In some instances, obtaining positional difference data between a first image and a second image includes: obtaining image feature data of the first image and the second image through training parameters, wherein the training parameters are used to obtain image feature data of each image in a model to be trained; determining a positional relationship between the first image and the second image based on the image feature data; and determining the positional difference data based on the positional relationship.
[0083] Among them, image feature data, also called image features, refers to the attributes that describe image information, such as image color, texture, spatial position relationship, etc.
[0084] Positional relationship refers to the positional relationship between corresponding pixels in an image. In this positional relationship, the positional relationship between corresponding pixels can also make the images have a positional relationship, such as rotation, translation, scaling, etc. between the overall images. The overall image may not have a positional relationship, and each pixel will have its own independent position adjustment, and the overall image deformation is not uniform.
[0085] For example, according to the foregoing, the server performs convolution operations, pooling operations (convolution operations and pooling operations may also be referred to as downsampling operations) and upsampling operations on the first image and the second image through the training parameters preset in the initial model, thereby obtaining image feature data of each image. Based on the image feature data, corresponding pixels in the two images may be determined, such as the same or corresponding pixels may be determined based on the similarity of the image feature data.
[0086] Taking a group of identical or corresponding pixel points as an example, based on the coordinates of the pixel points in the first image, the coordinate relationship of the second image pixel points relative to the first image pixel points, that is, the coordinate deviation, is determined, thereby determining the coordinate deviations of multiple groups of identical or similar corresponding pixel points, thereby generating a coordinate deviation set, that is, the deformation field.
[0087] It should be noted that when a pixel point in the second image does not have a corresponding pixel point in the first image, such as there is an image difference between the first image and the second image, or the first image does not have a local image taken by the second image, the coordinate offset of the pixel point can be determined directly based on the positional relationship between the pixel points in the second image.
[0088] For multimodal images, since each image is in a different coordinate system, it is impossible to directly correspond to each other. It is necessary to switch the coordinates first so that multiple images have a unique coordinate system or a unique positioning standard before continuing the image processing operation. This process is also the process of image registration.
[0089] In some examples, acquiring the target object includes: stitching the first image and the second image according to the position difference data; and acquiring the target object according to the stitched image.
[0090] The stitched image, also known as a fused image, refers to images with overlapping parts (which may be obtained at different times, different viewing angles or different sensors) stitched together into a seamless panorama or high-resolution image. In this example, the overlapping part may be a target object.
[0091] In some examples, stitching the first image and the second image includes: processing the second image according to the position difference data; and stitching the first image and the processed second image to obtain a stitched image.
[0092] In some examples, the method 200 further includes: inputting the stitched image into a model to be trained.
[0093] For example, according to the above, if Figure 3 As shown, after the server 101 obtains the deformation field through the initial model, the server 101 processes the second image according to the deformation field through image interpolation, splices the first image and the processed image, and then the server 101 inputs the spliced image into the initial model.
[0094] It should be noted that since the image stitching method is a relatively mature technology in this field, it will not be described in detail here.
[0095] In addition, the execution process of processing the second image can also be performed in the initial model, so there is no need for the server to input the processed second image into the initial model, and the initial model directly performs image stitching based on the processed second image.
[0096] In some instances, obtaining a target object based on the stitched image includes: obtaining image feature data of the stitched image based on training parameters, where the training parameters are used to obtain image feature data of each image in the model to be trained; and segmenting the target object from the stitched image based on the image feature data of the stitched image.
[0097] For example, according to the above, if Figure 3 As shown, after the initial model obtains the spliced image, the server 101 performs convolution operation, pooling operation (convolution operation and pooling operation may also be referred to as downsampling operation) and upsampling operation on the first image and the second image through the training parameters preset in the initial model, thereby obtaining the image feature data of the spliced image. The target object may be segmented according to the feature data corresponding to each pixel in the image feature data of the spliced image, such as clustering the pixels according to the feature data of the pixels to segment the image.
[0098] It should be noted that the training parameters for obtaining the image feature data of the stitched image are the same as the training parameters for obtaining the image feature data of the first image and the second image.
[0099] 203: Train the model to be trained according to the target object and the position difference data.
[0100] In some instances, training the model to be trained includes: obtaining a reference image of the target object; processing a second image based on position difference data; determining a first loss function based on the processed image and the first image; determining a second loss function based on the target object and the reference image of the target object; training the model to be trained based on the first loss function and the second loss function to generate an image processing model.
[0101] Among them, the loss function reflects the difference between the predicted results and the actual results.
[0102] The reference image of the target object refers to an actual image of the target object, such as an actual image of a vehicle.
[0103] For example, according to the above, if Figure 3 As shown, the server 101 obtains a reference image of the target object from its database or local or other server nodes, and the server 101 uses the similarity between the processed second image and the first image as the first loss function. The server 101 determines the second loss function based on the target object output by the initial model and its reference image, adjusts the training parameters based on the first loss function and the second loss function, and trains the initial model, thereby finally generating an image processing model.
[0104] It should be noted that after adjusting the training parameters, the initial model has started the optimization process, and the above steps 201-203 are repeated to continue optimizing the model, that is, iterating the training parameters until the number of iterations reaches the number threshold, or until the first loss function and the second loss function reach the preset range, then the model training is completed. Since the iterative process is a process of repeating the above implementation content, it will not be repeated here.
[0105] In addition, the method for determining the loss function is already a relatively mature technology in this field and will not be repeated here. The loss function can be a regression loss function or a classification loss function.
[0106] In some instances, determining the first loss function includes determining the first loss function based on gradient information of pixels in the processed image and gradient information of pixels in the first image.
[0107] Among them, the first loss function NGF (normalized gradient field) is determined according to the following formula 1):
[0108]
[0109] Among them, X, Y are the length and width of the image, δ is a differentiable function, which is a very small floating point value, x is the gradient of the pixel in the x direction, y is the gradient of the pixel in the y direction, and R x is the gradient of the pixel in the first image in the x direction, R y is the gradient of the pixel in the first image in the y direction, F(φ) x is the gradient of the pixel in the second image in the x direction after processing, F(φ) y is the gradient of the pixel in the second image in the y direction after processing.
[0110] It should be noted that by determining the first loss function through gradient information, the positional relationship of the second image relative to the first image in the multimodal image can be better processed, and the processing can be more accurate.
[0111] In some instances, training the model to be trained includes: determining the sum of a first loss function and a second loss function; adjusting training parameters based on the sum, thereby training the model to be trained until the training results meet expectations, and the training parameters are used to obtain image feature data for each image in the model to be trained.
[0112] Among them, the expected situation refers to the estimated demand, which may include: training the model by iterative training parameters until the number of iterations reaches a threshold, or until the first loss function and the second loss function reach within a preset range, or until the sum of the first loss function and the second loss function reaches within a preset range.
[0113] For example, according to the above, if Figure 3 As shown, after obtaining the values of the first loss function and the second loss function, the server 101 determines the sum of the two values, and adjusts the training parameters through a parameter optimization algorithm based on the sum, thereby completing the first training of the model, repeating the above implementation content, and training the model multiple times until the training parameters make the first loss function and the second loss function reach a preset range, or the number of training times (i.e., the number of iterations) reaches a threshold number, and finally a trained model is generated, which can be used for processing.
[0114] It should be noted that the parameter optimization algorithm is a relatively mature technology in this field and will not be described in detail here. It can be a gradient descent algorithm, etc.
[0115] In some instances, image feature data is obtained, including: downsampling the image according to training parameters and sampling levels; upsampling the image processed at the lowest sampling level according to training parameters and sampling levels; for each sampling level, splicing the upsampled image and the downsampled image of the sampling level as the data to be upsampled at the sampling level; and using the data to be upsampled corresponding to the highest sampling level as the image feature data.
[0116] The sampling level, which may also be referred to as the sampling level, refers to the number of times the image or image feature is downsampled or upsampled. The sampling level includes at least two sampling levels, such as 3 times and 5 times.
[0117] Downsampling refers to the process of reducing the size of an image to obtain the depth features of the image, which may include convolution and pooling operations.
[0118] Upsampling refers to enlarging an image, which enlarges the downsampled image and may include a deconvolution operation.
[0119] For example, according to the foregoing, the model to be trained can be a model based on the U-NET network structure. When acquiring image feature data, the image is first downsampled, such as performing a convolution operation and a pooling operation on the image, completing such a downsampling operation, that is, completing a sampling level, and performing the operation according to the preset number of sampling levels until the final downsampling result is obtained. Upsampling is started according to the final downsampling result, such as a deconvolution operation, completing a deconvolution operation, obtaining a deconvolution result, as a sampling level for upsampling, and performing the operation according to the preset number of sampling levels to obtain the final upsampling result. In the process of upsampling, after obtaining the upsampling result for the first time, it is necessary to first perform image splicing of the upsampling result and the downsampling result of this time, and continue to perform the next upsampling operation on the spliced image (i.e., the data to be upsampled), and each subsequent upsampling operation is upsampled for the spliced image until the spliced image of the top level is finally obtained as the image feature data.
[0120] During the upsampling process, each downsampling result can be used as the final downsampling result to perform an upsampling operation. The operation process is the same as described above and will not be repeated here. At this time, there are multiple sampling levels, and there may be multiple upsampling results for the same sampling level. When there are at least two or more sampling results (which may include downsampling results and upsampling results) at the level before the upsampling process is performed at the level, it is necessary to superimpose each sampling result of the level, and perform an upsampling operation based on the superimposed result until the superimposed result of the final level, i.e., the spliced image, is obtained as the final upsampling result.
[0121] It should be noted that after the downsampling operation and the upsampling operation, the acquired image feature data includes multiple levels of feature data processed at multiple levels, that is, the richness of features at different levels is maintained. The number of downsampling times can be increased or decreased to meet different data requirements, and the number of upsampling times can also be increased or decreased to meet different data requirements.
[0122] The embodiment of the present application, by using artificial intelligence to implement the technical solution, has a very large improvement over the traditional algorithm, both in terms of robustness and algorithm operation efficiency. The embodiment of the present application constructs a network to achieve parameter sharing, performs internal iteration, and outputs the results of two tasks with one network. After experimental comparison, the accuracy of the registration of the embodiment of the present application is within the error range of 2.7 pixels, and the segmentation accuracy can also reach the accuracy of a single segmentation task. Since the embodiment of the present application is an end-to-end network, the operating efficiency is significantly improved and the running time is greatly reduced.
[0123] Figure 4A This is a flowchart of an image processing method according to an exemplary embodiment of the present application. The method 400A provided in the embodiment of the present application is executed by a second device, such as a server, and the method 400A includes the following steps:
[0124] 401: Acquire a first image and a second image, wherein the first image is used as a reference image and the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object.
[0125] 402: Input the first image and the second image into an image processing model to obtain a target object, where the obtained target object has image features of the first image and the second image.
[0126] It should be noted that since steps 401-402 have been described in detail in the previous text, they will not be repeated here.
[0127] It is only explained that the method 400 is similar to the specific implementation of the method 200 described above. In addition to being executed by a server, the method 400 can also be directly executed by a terminal with powerful intelligent functions, such as a computer in a smart terminal. The device executing the method 400 can be deployed with a display component, such as a display screen, and the obtained final image result can be directly displayed through the display screen for users to view.
[0128] For example, as shown in FIG4B, the computer 403 may also send the first image "CT image of a certain muscle tissue" and the second image "ultrasound image of a certain muscle tissue" to the server 404. After the server 404 obtains the two images, they are input into the local image processing model, which is obtained after training with the above method 200. The model outputs the final target object, i.e., the segmented image, such as the image of a certain muscle tissue, which combines the image elements and image features of the first image and the second image, i.e., the image is obtained from the spliced image after the first image and the second image are spliced. The server 404 returns the image to the computer 403 for the user to view.
[0129] In some examples, the method 400A further includes: acquiring position difference data between the first image and the second image; wherein acquiring the target object includes: acquiring the target object according to the difference data.
[0130] It should be noted that since this example has been explained in detail in the previous article, it will not be repeated here.
[0131] In some examples, acquiring the target object includes: processing the second image according to the position difference data; splicing the first image and the processed second image to obtain a spliced image; and segmenting the target object from the spliced image.
[0132] It should be noted that since this example has been explained in detail in the previous article, it will not be repeated here.
[0133] After the target object is acquired through the trained image processing model, in order to ensure whether the acquired target object is accurate, the acquired target object can also be verified.
[0134] The verification method may include: verifying the acquired target object according to the third image, the first image, the second image and the third image being different images for the same target object; when the verification result is passed, determining that the acquired target object is the final target object.
[0135] The third image is an image of the target object, but the type or modality of the third image is different from the first image and the second image, but can belong to the type of the first image and the second image, such as a medical image: a magnetic resonance image, a CT image, an ultrasound image, an X-ray image, etc. It can also be a monitoring image.
[0136] For example, according to the above, after acquiring an image of a muscle tissue, the server can verify whether the currently acquired image of the muscle tissue is accurate through the "magnetic resonance image of a muscle tissue". By comparing the image features of the two images, when the comparison result is within the comparison threshold, it is determined that the acquired image of the muscle tissue is accurate, otherwise it is inaccurate.
[0137] Figure 5A A flowchart of another image processing method provided by another exemplary embodiment of the present application is provided. The method 500A provided by the embodiment of the present application is executed by a second device, such as a server, and the method 500A includes the following steps:
[0138] 501: Acquire a first image and a second image, wherein the first image is used as a reference image and the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object.
[0139] 502: Input the first image and the second image into the model to be trained, obtain position difference data between the first image and the second image through the first sub-model in the model to be trained, obtain the target object through the position difference data and the second sub-model in the model to be trained, the obtained target object has the image features of the first image and the second image, and the training parameters of the first sub-model and the second sub-model are the same.
[0140] 503: Train the training parameters according to the target object and the position difference data, so as to train the model to be trained.
[0141] It should be noted that since the specific implementation of steps 501-503 is similar to the specific implementation of method 200 described above, they will not be repeated here.
[0142] It is only explained that the method 500 can be executed by a server or directly by a terminal with powerful intelligent functions, such as a computer in an intelligent terminal. The device executing the method 500 can be deployed with a display component, such as a display screen, and the obtained final image result can be directly displayed through the display screen for users to view.
[0143] In addition, for the first sub-model and the second sub-model, the specific implementation process of obtaining the position difference data is used as the first sub-model, and the specific implementation process of obtaining the target object is used as the second sub-model. The two sub-models share parameters. That is to say, when training the model, the shared parameters are trained to complete the training of the two sub-models, thereby completing the training process of the entire model. The training process is similar to the training process described above, and will not be repeated here.
[0144] For example, as shown in Figure 5B, the server 504 obtains multiple groups of training samples, each group of training samples may include a reference image (i.e., a first image), and an image to be calibrated (a second image). At the same time, the server also obtains a class of samples, which are reference images of the target object corresponding to each group of training samples. The server 504 inputs the first image and the second image into the model to be trained, i.e., the initial model, determines the position difference data of the second image relative to the first image through its first sub-model, processes the second image according to the position difference data, splices the processed second image with the first image, and segments the target object in the spliced image through its second sub-model. Thus, the loss function between the target object and the reference image of the target object, and the loss function between the processed second image and the first image are determined. The initial parameters of the model, i.e., the training parameters, are adjusted according to the sum of the two loss functions. After the adjustment, the model is continued to be trained, and the training parameters are adjusted to implement the iteration of the training parameters until the adjusted training parameters make the sum of the loss functions obtained by the model within the expected numerical range, or the number of iterations reaches the expected number of times, etc. Thus, the first sub-model and the second sub-model are trained, and the final image processing model is generated.
[0145] Fig. 6A A flowchart of another image processing method provided by another exemplary embodiment of the present application is provided. The method 600A provided by the embodiment of the present application is executed by a second device, such as a server, and the method 600A includes the following steps:
[0146] 601: Acquire a first medical image and a second medical image, where the first medical image and the second medical image are different modality images of the same abnormal area.
[0147] 602: Input the first medical image and the second medical image into an image processing model to obtain an abnormal region in the spliced first medical image and the second medical image, where the obtained abnormal region has image features of the first medical image and the second medical image.
[0148] 603: Provide an image of the abnormal area, with the first medical image serving as a reference image and the second medical image serving as an image to be calibrated.
[0149] It should be noted that since the specific implementation of steps 601-603 is similar to the specific implementation of method 200 described above, they will not be repeated here.
[0150] It is only explained that the method 600 can be executed by a server or directly by a terminal with powerful intelligent functions, such as a computer in a smart terminal. The device executing the method 600 can be deployed with a display component, such as a display screen, and the obtained final image result can be directly displayed through the display screen for users to view.
[0151] The first medical image, which may also be referred to as a medical image, refers to an image of the internal tissue of a human body or a part of a human body obtained in a non-invasive manner, and the second medical image, which may also be referred to as a medical image, refers to an image of the internal tissue of a human body or a part of a human body obtained in a non-invasive manner, such as a magnetic resonance image, a CT image, an ultrasound image, an X-ray image, etc.
[0152] An abnormal area is a part of the body where something abnormal occurs.
[0153] The first medical image and the second medical image may be preset as a reference image and an image to be calibrated, respectively, or the reference image and the image to be calibrated may be specified by an image processing model.
[0154] For example, as shown in FIG6B, when performing a physical examination, such as when inspecting an abnormal liver area, multiple modal images of the liver are taken, and the computer 604 can also send the liver CT image and the liver magnetic resonance image to the server 605. After the server 605 obtains these two images, they are input into the local image processing model, which is obtained after training with the above method 200. The model outputs an image of the abnormal liver area, which combines the image elements and image features of the liver CT image and the liver magnetic resonance image, that is, the image is obtained from the spliced image after the liver CT image and the liver magnetic resonance image are spliced. The server 605 returns the image to the computer 604 for the user to view, such as for the doctor to view the abnormal state and location.
[0155] Specifically, obtaining the abnormal area in the stitched first medical image and the second medical image includes: obtaining position difference data of the first medical image and the second medical image through an image processing model; processing the second medical image according to the difference data through the image processing model; stitching the first image and the processed second image to obtain a stitched image through the image processing model; and segmenting the abnormal area from the stitched image through the image processing model.
[0156] Since the specific implementation method of image segmentation has been described in detail above, it will not be repeated here.
[0157] In order to provide users with a better service experience, different services can be performed according to different users.
[0158] The method of providing different services to different users may include: obtaining a user identifier, determining a user level or user authority based on the user identifier; and obtaining and providing an image of an abnormal area corresponding to the user based on the user level or user authority.
[0159] The user level refers to the level of the user, such as ordinary user, advanced user, and VIP user, etc. It should be understood that users of different levels have different service rights.
[0160] User rights refer to the service rights that users can use, such as obtaining and providing images of abnormal liver areas.
[0161] For example, according to the foregoing, the server obtains medical images, such as liver CT images and liver magnetic resonance images, the identifier of the user, such as the user ID, or the level corresponding to the user ID, such as a senior user, or the authority corresponding to the user, such as, it can obtain segmented images of multiple modal images. When the corresponding user level or user authority can obtain the liver abnormal area image, the server can fuse the liver CT image and the liver magnetic resonance image according to the image processing model to obtain the liver abnormal area image, and provide the liver abnormal area image to the user, such as sending it to the user's terminal, email client, etc. When the user does not have this authority, the server may not perform the operation of obtaining the liver abnormal area image and does not provide this service to the user.
[0162] In addition, the method 600 further includes: determining the usage fee of the image processing service, the waiting time of the image processing service, the payment method of the usage fee and / or the method of providing the image acquisition according to the user level or the user authority.
[0163] For example, according to the above, the server can also determine the usage fee for providing the abnormal liver region image according to the user level, such as a senior user, such as a certain fee discount for a senior user, and the usage fee is 80% of the original fee, and the usage fee for an ordinary user is the original fee. The senior user can pay by installment, and the time for the senior user to get the abnormal liver region image can be one week, and the time for the ordinary user can be 2 weeks. The senior user can get the abnormal liver region image by receiving express delivery, and the abnormal liver region image can be received by mail. The ordinary user can go to the image extraction place in person to get the abnormal liver region image.
[0164] Figure 7 A flowchart of another image processing method provided by another exemplary embodiment of the present application is provided. The method 700 provided by the embodiment of the present application is executed by a second device, such as a server, and the method 700 includes the following steps:
[0165] 701: Acquire a first medical image and a second medical image, where the first medical image and the second medical image are different modality images of the same abnormal area.
[0166] 702: Input the first medical image and the second medical image into the image processing model, obtain the abnormal region in the spliced first medical image and the second medical image, and determine the location of the abnormal region, wherein the acquired abnormal region has image features of the first medical image and the second medical image.
[0167] 703: Provide an image and a position of the abnormal area, with the first medical image serving as a reference image and the second medical image serving as an image to be calibrated.
[0168] It should be noted that since the specific implementation of steps 701-703 is similar to the specific implementation of method 200 described above, they will not be repeated here.
[0169] It is only explained that the method 700 can be executed by a server or directly by a terminal with powerful intelligent functions, such as a computer in an intelligent terminal. The device executing the method 700 can be deployed with a display component, such as a display screen, and the obtained final image result can be directly displayed through the display screen for users to view.
[0170] For example, according to the above, during the operation, the computer can also send the liver CT image taken before the operation and the liver ultrasound image taken during the operation to the server. After the server obtains these two images, they are input into the local image processing model, which is obtained after training by the above method 200. The model outputs the liver abnormal area image and the position of the liver abnormal area in the ultrasound image, such as coordinates. The liver abnormal area image combines the image elements and image features of the liver CT image and the liver ultrasound image, that is, the image is obtained from the spliced image after the liver CT image and the liver ultrasound image are spliced. The server returns the image and the image position to the computer for the user to view, such as for the doctor to view the abnormal state and position, so as to perform surgical navigation and track the abnormal area.
[0171] Fig. 8A A flowchart of another image processing method provided by another exemplary embodiment of the present application is provided. The method 800A provided by the embodiment of the present application is executed by a second device, such as a server, and the method 800A includes the following steps:
[0172] 801: Acquire a first monitoring image and a second monitoring image, where the first monitoring image and the second monitoring image come from different monitoring devices, the first monitoring image serves as a reference image, and the second monitoring image serves as an image to be calibrated, and the first monitoring image and the second monitoring image are different images of the same monitoring target object.
[0173] 802: Input the first monitoring image and the second monitoring image into an image processing model to obtain a monitoring target object, where the obtained monitoring target object has image features of the first monitoring image and the second monitoring image.
[0174] 803: Identify the information to be found in the monitoring target object.
[0175] It should be noted that since the specific implementation of steps 801-803 is similar to the specific implementation of method 200 described above, they will not be repeated here.
[0176] It is only explained that the method 800 can be executed by a server or directly by a terminal with powerful intelligent functions, such as a computer in a smart terminal. The device executing the method 800 can be deployed with a display component, such as a display screen, and the obtained final image result can be directly displayed through the display screen for users to view.
[0177] The first monitoring image refers to an image taken at different times and / or angles at the same location, such as a person image. The second monitoring image refers to an image taken at different times and / or angles at the same location, such as a person image.
[0178] The information to be searched refers to the information on the monitored target object, such as the license plate of a vehicle, the color of a person's clothes, a sign, etc.
[0179] For example, as shown in FIG8B, from a certain location in a shopping mall, at a certain time, the video images captured by two cameras 804 and 805 at the location are obtained, and the two cameras 804 and 805 are respectively set at different positions of the location, such as the left and right sides of the same height on the wall. The same person is shown in the video screen. The person is displayed in the video screen at different angles. The camera sends the video image, i.e., the video screen, to the server 806 in the form of a first image and a second image, respectively, such as the image captured by the camera 804 is the first image, and the image captured by the camera 805 is the second image. After the server 806 obtains the two images, it is input into the local image processing model, which is obtained after training with the above method 200. The model outputs a segmented image, such as the image of the person, which combines the image elements and image features of the first image and the second image, that is, the image is obtained from the spliced image after the first image and the second image are spliced, so that the person is clearer than in a single image. Server 806 identifies the image and obtains its detailed information, such as the face of the person, the brand of the person's clothes, etc. Server 806 returns the identified information and the image to computer 807 for users to view, such as for investigators to view and analyze the image, for investigators to search for clues based on the identified information, etc.
[0180] Fig. 9 A flowchart of a video processing method provided by another exemplary embodiment of the present application is provided. The method 900 provided by the embodiment of the present application is executed by a second device, such as a server, and the method 900 includes the following steps:
[0181] 901: Acquire a first surveillance video and a second surveillance video, where the first surveillance video and the second surveillance video are from different surveillance devices.
[0182] 902: Acquire a first surveillance image from a first surveillance video, and acquire a second surveillance image corresponding to the first surveillance video from a second surveillance video, wherein the first surveillance image serves as a reference image, and the second surveillance image serves as an image to be calibrated, and the acquired first surveillance image and second surveillance image are different images of the same surveillance target object.
[0183] 903: Input the first monitoring image and the second monitoring image into the image processing model to obtain a monitoring target object, where the monitoring target object has image features of the first monitoring image and the second monitoring image.
[0184] 904: Identify the information to be found in the monitoring target object.
[0185] It should be noted that since the specific implementation of steps 901-904 is similar to the specific implementation of method 200 described above, they will not be repeated here.
[0186] It is only explained that the method 900 can be executed by a server or directly by a terminal with powerful intelligent functions, such as a computer in a smart terminal. The device executing the method 900 can be deployed with a display component, such as a display screen, and the obtained final image result can be directly displayed through the display screen for users to view.
[0187] The first surveillance video refers to a video shot at different times or angles at the same location, such as a video containing a person's image. The second surveillance video refers to a video shot at different times or angles at the same location, such as a video containing a person's image.
[0188] For example, according to the above description, at a certain location on a highway section, at a certain time, a video captured by two cameras at the location is obtained, and the two cameras are respectively set at different positions of the location, such as the left and right sides of the same height on the mast. The same vehicle is shown in the video. The vehicle is displayed in the video at different angles. The camera sends the video to the server. After the server 806 obtains the two videos, it obtains corresponding images from the two videos according to time, such as the first camera image and the second camera image at 20:00:00 in the evening as corresponding images. The server inputs the two images into the local image processing model in the form of the first image and the second image respectively. The model is obtained after training with the above method 200. The model outputs a segmented image, such as an image of the vehicle, which combines the image elements and image features of the first image and the second image, that is, the image is obtained from the spliced image after the first image and the second image are spliced, so that the vehicle is clearer than in a single image and a single video. The server identifies the image and obtains its detailed information, such as the license plate of the vehicle. The server returns the identified license plate information and the image to the computer for the user to view, such as for investigators to view and analyze the image, for investigators to search for clues to the identified information, etc.
[0189] Fig.10 A schematic diagram of the structural framework of an image processing device provided by an exemplary embodiment of the present application. The device 1000 can be applied to a second device, for example, a server, and the device 1000 includes an acquisition module 1001, an input module 1002, and a training module 1003. The functions of each module are described in detail below:
[0190] The acquisition module 1001 is used to acquire a first image and a second image, wherein the first image is used as a reference image and the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object.
[0191] The input module 1002 is used to input the first image and the second image into the model to be trained, obtain the position difference data of the first image and the second image and the target object, and the obtained target object has the image features of the first image and the second image.
[0192] The training module 1003 is used to train the model to be trained according to the target object and the position difference data.
[0193] In some instances, the input module 1002 includes: a first acquisition unit, used to acquire image feature data of the first image and the second image through training parameters, the training parameters are used to acquire image feature data of each image in the model to be trained; a first determination unit, used to determine the positional relationship between the first image and the second image based on the image feature data; and determine the positional difference data based on the positional relationship.
[0194] In some examples, the input module 1002 includes: a stitching unit, configured to stitch the first image and the second image according to the position difference data; and a first acquisition unit, configured to acquire the target object according to the stitched image.
[0195] In some examples, the stitching unit is used to process the second image according to the position difference data; and stitch the first image and the processed second image to obtain a stitched image.
[0196] In some instances, the first acquisition unit is used to acquire image feature data of the stitched image based on training parameters, and the training parameters are used to acquire image feature data of each image in the model to be trained; based on the image feature data of the stitched image, the target object is segmented from the stitched image.
[0197] In some instances, the input module 1002 is further used to input the stitched image into the model to be trained.
[0198] In some instances, the training module 1003 includes: a second acquisition unit for acquiring a reference image of the target object; a processing unit for processing the second image based on the position difference data; a second determination unit for determining a first loss function based on the processed image and the first image; determining a second loss function based on the target object and the reference image of the target object; and a training unit for training the model to be trained based on the first loss function and the second loss function to generate an image processing model.
[0199] In some instances, the second determination unit is used to determine the first loss function based on gradient information of pixel points in the processed image and gradient information of pixel points in the first image.
[0200] In some instances, the training unit is used to determine the sum of the first loss function and the second loss function; based on the sum, the training parameters are adjusted to train the model to be trained until the training results meet expectations, and the training parameters are used to obtain image feature data for each image in the model to be trained.
[0201] In some examples, the image feature data includes multiple levels of feature data processed at multiple levels.
[0202] In some instances, the first acquisition unit is used to downsample the image according to the training parameters and the sampling level; upsample the image processed at the lowest sampling level according to the training parameters and the sampling level; for each sampling level, splice the upsampled image and the downsampled image of the sampling level as the data to be upsampled at the sampling level; and use the data to be upsampled corresponding to the highest sampling level as image feature data.
[0203] In some examples, the sampling hierarchy includes at least two sampling hierarchies.
[0204] Another exemplary embodiment of the present application provides another image processing device. The device 1100 can be applied to a second device for execution, such as a server; the device 1100 includes: an acquisition module 1101 and an input module 1102. The functions of each module are described in detail below:
[0205] The acquisition module 1101 is used to acquire a first image and a second image, wherein the first image is used as a reference image and the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object.
[0206] The input module 1102 is used to input the first image and the second image into the image processing model to obtain a target object, where the obtained target object has image features of the first image and the second image.
[0207] In some examples, the acquisition module 1101 is further used to acquire position difference data between the first image and the second image; wherein the input module 1102 is used to acquire the target object according to the difference data.
[0208] In some instances, the input module 1102 includes: a processing unit for processing the second image according to the position difference data; a stitching unit for stitching the first image and the processed second image to obtain a stitched image; and a segmentation unit for segmenting the target object from the stitched image.
[0209] In some instances, the device 1100 also includes: a verification module, used to verify the acquired target object based on the third image, the first image, the second image and the third image are different images for the same target object; when the verification result is passed, it is determined that the acquired target object is the final target object.
[0210] Another exemplary embodiment of the present application provides another image processing device. The device 1200 can be applied to a second device for execution, such as a server; the device 1200 includes: an acquisition module 1201, an input module 1202 and a training module 1203. The functions of each module are described in detail below:
[0211] The acquisition module 1201 is used to acquire a first image and a second image, wherein the first image is used as a reference image and the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object.
[0212] Input module 1202 is used to input the first image and the second image into the model to be trained, obtain the position difference data between the first image and the second image through the first sub-model in the model to be trained, and obtain the target object through the position difference data and the second sub-model in the model to be trained. The obtained target object has the image features of the first image and the second image, and the training parameters of the first sub-model and the second sub-model are the same.
[0213] The training module 1203 is used to train the training parameters according to the target object and the position difference data, so as to train the model to be trained.
[0214] Another exemplary embodiment of the present application provides another image processing device. The device 1300 can be applied to a second device for execution, such as a server; the device 1300 includes: an acquisition module 1301, an input module 1302, and a providing module 1303. The functions of each module are described in detail below:
[0215] The acquisition module 1301 is used to acquire a first medical image and a second medical image, where the first medical image and the second medical image are different modality images of the same abnormal area.
[0216] The input module 1302 is used to input the first medical image and the second medical image into the image processing model to obtain the abnormal area in the spliced first medical image and the second medical image, and the obtained abnormal area has the image features of the first medical image and the second medical image.
[0217] A module 1303 is provided, which is used to provide an image of the abnormal area, wherein the first medical image is used as a reference image and the second medical image is used as an image to be calibrated.
[0218] In some instances, the input module 1302 includes: an acquisition unit, used to acquire position difference data between the first medical image and the second medical image through an image processing model; a processing unit, used to process the second medical image according to the difference data through the image processing model; a stitching unit, used to stitch the first image and the processed second image through the image processing model to obtain a stitched image; and a segmentation unit, used to segment an abnormal area from the stitched image through the image processing model.
[0219] In some examples, the device 1300 further includes: a determination module that obtains a user identifier and determines a user level or user authority based on the user identifier; and obtains and provides an image of an abnormal area corresponding to the user based on the user level or user authority.
[0220] In some examples, the determination module is further used to determine the usage fee of the image processing service, the waiting time of the image processing service, the payment method of the usage fee and / or the method of obtaining the image according to the user level or user authority.
[0221] Another exemplary embodiment of the present application provides another image processing device. The device 1400 can be applied to a second device for execution, such as a server; the device 1400 includes: an acquisition module 1401, an input module 1402, and a providing module 1403. The functions of each module are described in detail below:
[0222] The acquisition module 1401 is used to acquire a first medical image and a second medical image, where the first medical image and the second medical image are different modality images of the same abnormal area.
[0223] The input module 1402 is used to input the first medical image and the second medical image into the image processing model, obtain the abnormal area in the spliced first medical image and the second medical image, and determine the location of the abnormal area. The acquired abnormal area has the image features of the first medical image and the second medical image.
[0224] A providing module 1403 is used to provide an image and a position of an abnormal area, with the first medical image being used as a reference image and the second medical image being used as an image to be calibrated.
[0225] Another exemplary embodiment of the present application provides another image processing device. The device 1500 can be applied to a second device for execution, such as a server; the device 1500 includes: an acquisition module 1501, an input module 1502 and a recognition module 1503. The functions of each module are described in detail below:
[0226] The acquisition module 1501 is used to acquire a first monitoring image and a second monitoring image, where the first monitoring image and the second monitoring image come from different monitoring devices, the first monitoring image serves as a reference image, and the second monitoring image serves as an image to be calibrated, and the first monitoring image and the second monitoring image are different images of the same monitoring target object.
[0227] The input module 1502 is used to input the first monitoring image and the second monitoring image into the image processing model to obtain a monitoring target object, and the obtained monitoring target object has image features of the first monitoring image and the second monitoring image.
[0228] The identification module 1503 is used to identify the information to be found in the monitoring target object.
[0229] Another exemplary embodiment of the present application provides a video processing device. The device 1600 can be applied to a second device for execution, such as a server; the device 1600 includes: an acquisition module 1601, an input module 1602 and an identification module 1603. The functions of each module are described in detail below:
[0230] The acquisition module 1601 is used to acquire a first surveillance video and a second surveillance video, where the first surveillance video and the second surveillance video are from different surveillance devices.
[0231] The acquisition module 1601 is used to acquire a first surveillance image from a first surveillance video, and to acquire a second surveillance image corresponding to the first surveillance video from a second surveillance video. The first surveillance image serves as a reference image, and the second surveillance image serves as an image to be calibrated. The first surveillance image and the second surveillance image are different images of the same surveillance target object.
[0232] The input module 1602 is used to input the first monitoring image and the second monitoring image into the image processing model to obtain a monitoring target object, and the obtained monitoring target object has image features of the first monitoring image and the second monitoring image.
[0233] The identification module 1603 is used to identify the information to be found in the monitoring target object.
[0234] The above describes Fig.10 The internal functions and structure of the processing device 1000 shown in FIG. 1 are shown in FIG. 1 , in one possible design. Fig.10 The structure of the processing device 1000 shown can be implemented as a server, such as Fig.11 As shown, the device 1700 may include: a memory 1701 and a processor 1702;
[0235] Memory 1701, used for storing computer programs;
[0236] Processor 1702 is used to execute a computer program to: obtain a first image and a second image, the first image is used as a reference image, the second image is used as an image to be calibrated, and the first image and the second image are different images for the same target object; input the first image and the second image into a model to be trained, obtain position difference data between the first image and the second image and the target object, the obtained target object has image features of the first image and the second image; and train the model to be trained according to the target object and the position difference data.
[0237] In some instances, processor 1702 is specifically used to: obtain image feature data of the first image and the second image through training parameters, where the training parameters are used to obtain image feature data of each image in the model to be trained; determine a positional relationship between the first image and the second image based on the image feature data; and determine position difference data based on the positional relationship.
[0238] In some examples, the processor 1702 is specifically configured to: stitch the first image and the second image according to the position difference data; and acquire the target object according to the stitched image.
[0239] In some examples, the processor 1702 is specifically configured to: process the second image according to the position difference data; and stitch the first image and the processed second image to obtain a stitched image.
[0240] In some instances, processor 1702 is specifically used to: obtain image feature data of the stitched image according to training parameters, where the training parameters are used to obtain image feature data of each image in the model to be trained; and segment the target object from the stitched image according to the image feature data of the stitched image.
[0241] In some instances, the processor 1702 is further used to: input the stitched image into the model to be trained.
[0242] In some instances, processor 1702 is specifically used to: obtain a reference image of the target object; a processing unit is used to process the second image based on the position difference data; determine a first loss function based on the processed image and the first image; determine a second loss function based on the target object and the reference image of the target object; train the model to be trained based on the first loss function and the second loss function to generate an image processing model.
[0243] In some instances, the processor 1702 is specifically configured to determine a first loss function based on gradient information of pixels in the processed image and gradient information of pixels in the first image.
[0244] In some instances, processor 1702 is specifically used to: determine the sum of the first loss function and the second loss function; adjust the training parameters based on the sum, thereby training the model to be trained until the training results meet expectations, and the training parameters are used to obtain image feature data for each image in the model to be trained.
[0245] In some examples, the image feature data includes multiple levels of feature data processed at multiple levels.
[0246] In some instances, processor 1702 is specifically used to: downsample the image according to training parameters and sampling levels; upsample the image processed at the lowest sampling level according to training parameters and sampling levels; for each sampling level, splice the upsampled image and the downsampled image of the sampling level as the data to be upsampled at the sampling level; and use the data to be upsampled corresponding to the highest sampling level as image feature data.
[0247] In some examples, the sampling hierarchy includes at least two sampling hierarchies.
[0248] In addition, an embodiment of the present invention provides a computer storage medium, when the computer program is executed by one or more processors, causing the one or more processors to implement Figure 2 The steps of the image processing method in the method embodiment.
[0249] The internal functions and structure of the processing device 1100 are described above. In a possible design, the structure of the processing device 1100 may be implemented as a server. The device 1800 may include: a memory 1801 and a processor 1802;
[0250] Memory 1801, used for storing computer programs;
[0251] Processor 1802 is used to execute a computer program to: acquire a first image and a second image, the first image serving as a reference image, the second image serving as an image to be calibrated, the first image and the second image being different images for the same target object; input the first image and the second image into an image processing model to acquire the target object, the acquired target object having image features of the first image and the second image.
[0252] In some examples, the processor 1802 is further used to: obtain position difference data between the first image and the second image; wherein the processor 1802 is specifically used to: obtain the target object according to the difference data.
[0253] In some examples, the processor 1802 is specifically configured to: process the second image according to the position difference data; stitch the first image and the processed second image to obtain a stitched image; and segment the target object from the stitched image.
[0254] In some instances, processor 1802 is further used to: verify the acquired target object based on the third image, the first image, the second image and the third image are different images for the same target object; when the verification result is passed, it is determined that the acquired target object is the final target object.
[0255] In addition, an embodiment of the present invention provides a computer storage medium, and when a computer program is executed by one or more processors, the one or more processors are caused to implement the steps of the image processing method in the embodiment of method 400A.
[0256] The internal functions and structure of the processing device 1200 are described above. In a possible design, the structure of the processing device 1200 may be implemented as a server. The device 1900 may include: a memory 1901 and a processor 1902;
[0257] Memory 1901, used for storing computer programs;
[0258] Processor 1902 is used to execute a computer program to: obtain a first image and a second image, the first image serving as a reference image, the second image serving as an image to be calibrated, the first image and the second image being different images for the same target object; inputting the first image and the second image into a model to be trained, obtaining position difference data between the first image and the second image through a first sub-model in the model to be trained, obtaining a target object through the position difference data and the second sub-model in the model to be trained, the obtained target object having image features of the first image and the second image, and the training parameters of the first sub-model and the second sub-model being the same; training the training parameters according to the target object and the position difference data, thereby training the model to be trained.
[0259] In addition, an embodiment of the present invention provides a computer storage medium. When a computer program is executed by one or more processors, the one or more processors are caused to implement the steps of the image processing method in the embodiment of method 500A.
[0260] The internal functions and structure of the processing device 1300 are described above. In a possible design, the structure of the processing device 1300 may be implemented as a server. The device 2000 may include: a memory 2001 and a processor 2002;
[0261] Memory 2001, used for storing computer programs;
[0262] Processor 2002 is used to execute a computer program to: obtain a first medical image and a second medical image, the first medical image and the second medical image are different modality images for the same abnormal area; input the first medical image and the second medical image into an image processing model to obtain an abnormal area in the spliced first medical image and the second medical image, the acquired abnormal area has image features of the first medical image and the second medical image; provide an image of the abnormal area, the first medical image is used as a reference image, and the second medical image is used as an image to be calibrated.
[0263] In some instances, processor 2002 is specifically used to: obtain position difference data between a first medical image and a second medical image through an image processing model; process the second medical image according to the difference data through the image processing model; splice the first image and the processed second image to obtain a spliced image through the image processing model; and segment an abnormal area from the spliced image through the image processing model.
[0264] In some examples, the processor 2002 is further used to: obtain a user identification, determine a user level or user authority based on the user identification; and obtain and provide an image of an abnormal area corresponding to the user based on the user level or user authority.
[0265] In some examples, the processor 2002 is further used to determine the usage fee of the image processing service, the waiting time of the image processing service, the payment method of the usage fee and / or the method of obtaining the image according to the user level or user authority.
[0266] In addition, an embodiment of the present invention provides a computer storage medium. When a computer program is executed by one or more processors, the one or more processors are caused to implement the steps of the image processing method in the embodiment of method 600A.
[0267] The internal functions and structure of the processing device 1400 are described above. In a possible design, the structure of the processing device 1400 may be implemented as a server. The device 2100 may include: a memory 2101 and a processor 2102;
[0268] Memory 2101, used for storing computer programs;
[0269] Processor 2102 is used to execute a computer program to: obtain a first medical image and a second medical image, the first medical image and the second medical image are different modality images for the same abnormal area; input the first medical image and the second medical image into an image processing model, obtain the abnormal area in the spliced first medical image and the second medical image, and determine the location of the abnormal area, the acquired abnormal area has image features of the first medical image and the second medical image; provide an image and a location of the abnormal area, the first medical image is used as a reference image, and the second medical image is used as an image to be calibrated.
[0270] In addition, an embodiment of the present invention provides a computer storage medium. When a computer program is executed by one or more processors, the one or more processors are caused to implement the steps of the image processing method in the embodiment of method 700 .
[0271] The internal functions and structure of the processing device 1500 are described above. In a possible design, the structure of the processing device 1500 may be implemented as a server. The device 2200 may include: a memory 2201 and a processor 2202;
[0272] Memory 2201, used for storing computer programs;
[0273] Processor 2202 is used to execute a computer program to: obtain a first monitoring image and a second monitoring image, where the first monitoring image and the second monitoring image are from different monitoring devices, the first monitoring image is used as a reference image, the second monitoring image is used as an image to be calibrated, and the first monitoring image and the second monitoring image are different images of the same monitoring target object; input the first monitoring image and the second monitoring image into an image processing model to obtain the monitoring target object, the obtained monitoring target object has image features of the first monitoring image and the second monitoring image; and identify information to be found in the monitoring target object.
[0274] In addition, an embodiment of the present invention provides a computer storage medium, and when the computer program is executed by one or more processors, it causes the one or more processors to implement the steps of the image processing method in the embodiment of method 800A.
[0275] The internal functions and structure of the processing device 1600 are described above. In a possible design, the structure of the processing device 1600 may be implemented as a server. The device 2300 may include: a memory 2301 and a processor 2302;
[0276] Memory 2301, used for storing computer programs;
[0277] Processor 2302 is used to execute a computer program to: obtain a first surveillance video and a second surveillance video, the first surveillance video and the second surveillance video are from different surveillance devices; obtain a first surveillance image from the first surveillance video, and obtain a second surveillance image corresponding to the first surveillance video from the second surveillance video, the first surveillance image is used as a reference image, the second surveillance image is used as an image to be calibrated, and the first surveillance image and the second surveillance image are different images for the same surveillance target object; input the first surveillance image and the second surveillance image into an image processing model to obtain a surveillance target object, the acquired surveillance target object has image features of the first surveillance image and the second surveillance image; and identify information to be found in the surveillance target object.
[0278] In addition, an embodiment of the present invention provides a computer storage medium, and when a computer program is executed by one or more processors, the one or more processors are caused to implement the steps of the video processing method in the embodiment of method 900 .
[0279] Another exemplary embodiment of the present application provides another method for processing an image. The method 2400 provided in the embodiment of the present application is executed by a second device, such as a server, and the method 2400 includes the following steps:
[0280] 2401: Acquire a first medical image and a second medical image, where the first medical image and the second medical image are different modality images of the same abnormal area, the first medical image is used as a reference image, and the second medical image is used as an image to be calibrated.
[0281] 2402: Input the first medical image and the second medical image into the model to be trained, obtain the position difference data and the abnormal area between the first medical image and the second medical image, and the obtained abnormal area has the image features of the first image and the second image.
[0282] 2403: Train the model to be trained based on the abnormal area and position difference data.
[0283] It should be noted that since the specific implementation of steps 2401-2403 is similar to the specific implementation of method 200 described above, they will not be repeated here.
[0284] An exemplary embodiment of the present application provides an image processing device. The device 2500 can be applied to a second device, such as a server, and the device 2500 includes an acquisition module 2501, an input module 2502, and a training module 2503. The functions of each module are described in detail below:
[0285] The acquisition module 2501 is used to acquire a first medical image and a second medical image, where the first medical image and the second medical image are different modality images of the same abnormal area, the first medical image is used as a reference image, and the second medical image is used as an image to be calibrated.
[0286] The input module 2502 is used to input the first medical image and the second medical image into the model to be trained, obtain the position difference data and abnormal area of the first medical image and the second medical image, and the acquired abnormal area has the image features of the first image and the second image.
[0287] The training module 2503 is used to train the training model according to the abnormal area and position difference data.
[0288] The internal functions and structure of the processing device 2500 are described above. In a possible design, the structure of the processing device 2500 may be implemented as a server. The device 2600 may include: a memory 2601 and a processor 2602;
[0289] Memory 2601, used for storing computer programs;
[0290] Processor 2602 is used to execute a computer program to: obtain a first medical image and a second medical image, the first medical image and the second medical image are different modality images for the same abnormal area, the first medical image is used as a reference image, and the second medical image is used as an image to be calibrated; input the first medical image and the second medical image into a model to be trained, obtain position difference data between the first medical image and the second medical image and the abnormal area, the acquired abnormal area has image features of the first image and the second image; and train the model to be trained based on the abnormal area and the position difference data.
[0291] In addition, an embodiment of the present invention provides a computer storage medium. When the computer program is executed by one or more processors, the one or more processors implement the steps of the image processing method in the embodiment of method 2400.
[0292] Another exemplary embodiment of the present application provides another method for processing an image. The method 2700 provided in the embodiment of the present application is executed by a second device, such as a server, and the method 2700 includes the following steps:
[0293] 2701: Acquire a first medical image and a second medical image, where the first medical image and the second medical image are different modality images of the same abnormal area.
[0294] 2702: Input the first medical image and the second medical image into the image processing model to obtain an abnormal region in the spliced first medical image and the second medical image, where the obtained abnormal region has image features of the first medical image and the second medical image.
[0295] 2703: Obtain model data of digital virtual human.
[0296] 2704: Fusing the model data with the abnormal area so that the fused digital virtual human has the abnormal area.
[0297] It should be noted that since the specific implementation of steps 2701-2702 is similar to the specific implementation of method 200 described above, it will not be repeated here. Only steps 2703 and 2704 are described:
[0298] 2703: Obtain model data of digital virtual human.
[0299] Among them, digital virtual human refers to a three-dimensional human body model synthesized by simulating real human organs through digital technology. The three-dimensional human body model not only has the appearance of the human body and the appearance of various organs such as liver, heart, and kidneys, but also has the metabolic function of each organ, and can more realistically display the normal physiological state of the human body and various changes that occur.
[0300] The three-dimensional human body model is obtained by scanning the human body's various cross-sections with digital scanning equipment, obtaining a large amount of photographic data of the human body as the human body collection data. Based on the collection data and 3D modeling software, a 3D human body model is established, and the established model is rendered to generate the final three-dimensional human body model.
[0301] For example, according to the foregoing, the server can directly obtain the 3D human body model data of the preset digital virtual human from a model data platform, such as a model server.
[0302] 2704: Fusing the model data with the abnormal area so that the fused digital virtual human has the abnormal area.
[0303] For example, as described above, after acquiring the 3D human body model data, the server imports the image of the abnormal liver area into the 3D human body model data, thereby generating a three-dimensional human body model with the abnormal liver area, and can monitor the abnormal liver area in the three-dimensional human body model, and record the changes in the abnormal liver area and the change process.
[0304] In some examples, the method 2700 further includes: monitoring changes in abnormal areas in the fused digital virtual human.
[0305] For example, according to the above, the server can obtain the physical condition data of the human body with abnormal liver area, and these physical condition data can determine the physical changes of the human body, and thus can also determine the physical changes of the three-dimensional human body model, such as the number of white blood cells, the number of red blood cells, etc. The server inputs these physical condition data into the three-dimensional human body model, so that the three-dimensional human body model can simulate the human body change data of the human body with abnormal liver area, especially the change data of the abnormal liver area. The server can regularly monitor the change data of the abnormal liver area.
[0306] In some examples, the method 2700 further includes: obtaining changes in the abnormal area through an image processing model; and verifying the obtained changes based on the monitored changes.
[0307] For example, according to the foregoing, the server can obtain the change data of the abnormal liver area, determine the current condition of the human body to which the abnormal liver area belongs, and can predict the physical condition of the human body, especially the change data of the abnormal liver area. In addition, after the server records the change data of the abnormal liver area of the human body, when performing real-time physical examination of the human body to which the abnormal liver area belongs, especially when performing examination of the abnormal liver area, such as CT results detected by different equipment, or CT results detected by different hospitals, and the results are different, the server can compare the recorded change data of the abnormal liver area of the human body with the above multiple results, and select the result that matches the change data as the final real-time test result.
[0308] An exemplary embodiment of the present application provides an image processing device. The device 2800 can be applied to a second device, such as a server, and the device 2800 includes an acquisition module 2801, an input module 2802, and a fusion module 2803. The functions of each module are described in detail below:
[0309] The acquisition module 2801 is used to acquire a first medical image and a second medical image, where the first medical image and the second medical image are different modality images of the same abnormal area.
[0310] The input module 2802 is used to input the first medical image and the second medical image into the image processing model to obtain the abnormal area in the spliced first medical image and the second medical image, and the obtained abnormal area has the image features of the first medical image and the second medical image.
[0311] The acquisition module 2801 is used to acquire the model data of the digital virtual human.
[0312] The fusion module 2803 is used to fuse the model data with the abnormal area so that the fused digital virtual human has the abnormal area.
[0313] In some examples, the device 2800 further includes: a monitoring module for monitoring changes in abnormal areas in the fused digital virtual human.
[0314] In some examples, the acquisition module 2801 is further used to acquire changes in the abnormal area through an image processing model; the device 2800 also includes: a verification module, which is used to verify the acquired changes based on the monitored changes.
[0315] The internal functions and structure of the processing device 2800 are described above. In a possible design, the structure of the processing device 2800 may be implemented as a server. The device 2900 may include: a memory 2901 and a processor 2902;
[0316] Memory 2901, used for storing computer programs;
[0317] Processor 2902 is used to execute a computer program to: obtain a first medical image and a second medical image, where the first medical image and the second medical image are different modality images for the same abnormal area; input the first medical image and the second medical image into an image processing model to obtain an abnormal area in the spliced first medical image and the second medical image, where the obtained abnormal area has image features of the first medical image and the second medical image; obtain model data of a digital virtual person; and fuse the model data with the abnormal area so that the fused digital virtual person has the abnormal area.
[0318] In some instances, the processor 2902 is further used to monitor changes in abnormal areas in the fused digital virtual human.
[0319] In some instances, the processor 2902 is further used to: obtain changes in the abnormal area through an image processing model; and verify the obtained changes based on the monitored changes.
[0320] In addition, an embodiment of the present invention provides a computer storage medium. When the computer program is executed by one or more processors, the one or more processors implement the steps of the image processing method in the embodiment of method 2700.
[0321] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 201, 202, 203, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.
[0322] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0323] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0324] The present invention is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable multimedia data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable multimedia data processing device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.
[0325] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable multimedia data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0326] These computer program instructions may also be loaded onto a computer or other programmable multimedia data processing device so that a series of operational steps are executed on the computer or other programmable device to produce computer-implemented processing, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0327] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0328] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0329] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0330] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for processing an image, characterized in that: include: Acquire a first image and a second image, wherein the first image is used as a reference image, and the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object; Inputting the first image and the second image into the model to be trained, obtaining position difference data between the first image and the second image and the target object, wherein the obtained target object has image features of the first image and the second image; The model to be trained is trained according to the target object and the position difference data.
2. The method according to claim 1, characterized in that The step of acquiring position difference data between the first image and the second image includes: Obtaining image feature data of the first image and the second image through training parameters, wherein the training parameters are used to obtain image feature data of each image in the model to be trained; determining a positional relationship between the first image and the second image according to the image feature data; The position difference data is determined according to the position relationship.
3. The method according to claim 1, characterized in that The obtaining of the target object comprises: splicing the first image and the second image according to the position difference data; The target object is acquired according to the stitched image.
4. The method according to claim 3, characterized in that The stitching of the first image and the second image comprises: processing the second image according to the position difference data; The first image and the processed second image are stitched together to obtain the stitched image.
5. The method according to claim 3 or 4, characterized in that: The step of acquiring the target object according to the stitched image includes: Obtaining image feature data of the spliced image according to training parameters, wherein the training parameters are used to obtain image feature data of each image in the model to be trained; The target object is segmented from the spliced image according to the image feature data of the spliced image.
6. The method according to claim 4 or 5, characterized in that: The method further comprises: The spliced image is input into the model to be trained.
7. The method according to claim 1, characterized in that The training of the model to be trained includes: Acquire a reference image of the target object; processing the second image according to the position difference data; Determine a first loss function according to the processed image and the first image; Determining a second loss function according to the target object and a reference image of the target object; According to the first loss function and the second loss function, the model to be trained is trained to generate an image processing model.
8. The method according to claim 7, characterized in that The determining of the first loss function comprises: The first loss function is determined according to the gradient information of the pixel points in the processed image and the gradient information of the pixel points in the first image.
9. The method according to claim 7 or 8, characterized in that: The training of the model to be trained includes: Determining a sum of the first loss function and the second loss function; According to the sum, the training parameters are adjusted to train the model to be trained until the training result meets expectations, and the training parameters are used to obtain image feature data of each image in the model to be trained.
10. The method according to claim 2 or 5, characterized in that: The image feature data includes multiple levels of feature data processed at multiple levels.
11. The method according to claim 2 or 5, characterized in that: Get image feature data, including: Downsampling the image according to the training parameters and the sampling level; According to the training parameters and the sampling level, upsampling is performed on the image processed by the lowest sampling level; For each sampling level, the upsampled image and the downsampled image of the sampling level are spliced and used as the data to be upsampled at the sampling level; The up-sampled data corresponding to the top sampling level is used as the image feature data.
12. The method according to claim 11, characterized in that The sampling hierarchy includes at least two sampling hierarchies.
13. A method for processing an image, characterized in that: include: Acquire a first image and a second image, wherein the first image is used as a reference image, and the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object; The first image and the second image are input into an image processing model to obtain the target object, wherein the obtained target object has image features of the first image and the second image, wherein in a model training stage, the image processing model is used as a model to be trained, and the image processing model is obtained by training according to the method described in any one of claims 1 to 12.
14. The method according to claim 13, characterized in that The method further comprises: Acquire position difference data between the first image and the second image; Wherein, obtaining the target object includes: The target object is acquired according to the position difference data.
15. The method according to claim 14, characterized in that The obtaining of the target object comprises: processing the second image according to the position difference data; splicing the first image and the processed second image to obtain the spliced image; The target object is segmented from the spliced image.
16. The method according to claim 13, characterized in that The method further comprises: verifying the acquired target object according to the third image, wherein the first image, the second image and the third image are different images of the same target object; When the verification result is passed, it is determined that the acquired target object is the final target object.
17. An image processing system, characterized in that: include: a first device and a second device; The first device acquires a first image and a second image, wherein the first image is used as a reference image, the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object; sending the first image and the second image to the second device; The second device inputs the first image and the second image into an image processing model to obtain the target object, wherein the obtained target object has image features of the first image and the second image; wherein, in the model training stage, the image processing model is used as a model to be trained, and the image processing model is obtained by training according to the method described in any one of claims 1 to 12; The target object is sent to the first device.
18. A method for processing an image, characterized in that: include: Acquire a first image and a second image, wherein the first image is used as a reference image, and the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object; Inputting the first image and the second image into the model to be trained, obtaining position difference data between the first image and the second image through the first sub-model in the model to be trained, obtaining the target object through the position difference data and the second sub-model in the model to be trained, the obtained target object having image features of the first image and the second image, and the training parameters of the first sub-model and the second sub-model are the same; The training parameters are trained according to the target object and the position difference data, thereby training the model to be trained.
19. A method for processing an image, characterized in that: include: Acquire a first medical image and a second medical image, wherein the first medical image and the second medical image are different modality images of the same abnormal area; Input the first medical image and the second medical image into the image processing model, obtain the abnormal area in the spliced first medical image and the second medical image, and the acquired abnormal area has the image features of the first medical image and the second medical image; wherein, in the model training stage, the image processing model is used as the model to be trained, the first medical image is the first image, the second medical image is the second image, and the abnormal area is used as the target object, and the image processing model is obtained by training according to the method described in any one of claims 1 to 12; An image of the abnormal area is provided; the first medical image is used as a reference image, and the second medical image is used as an image to be calibrated.
20. The method according to claim 19, characterized in that The step of obtaining and stitching the abnormal area in the first medical image and the second medical image includes: Acquiring position difference data of the first medical image and the second medical image through the image processing model; Processing the second medical image according to the position difference data by the image processing model; splicing the first medical image and the processed second medical image through the image processing model to obtain the spliced image; The abnormal area is segmented from the spliced image by using the image processing model.
21. The method according to claim 19, characterized in that The method further comprises: Obtaining a user ID, and determining a user level or user authority based on the user ID; According to the user level or user authority, an image of the abnormal area corresponding to the user is acquired and provided.
22. The method according to claim 21, characterized in that The method further comprises: The use fee of the image processing service, the waiting time of the image processing service, the payment method of the use fee and / or the method of obtaining the image are determined according to the user level or the user authority.
23. A method for processing an image, characterized in that: include: Acquire a first medical image and a second medical image, wherein the first medical image and the second medical image are different modality images of the same abnormal area; Input the first medical image and the second medical image into the image processing model, obtain the abnormal area in the spliced first medical image and the second medical image, and determine the location of the abnormal area, wherein the acquired abnormal area has the image features of the first medical image and the second medical image; wherein, in the model training stage, the image processing model is used as the model to be trained, the first medical image is used as the first image, the second medical image is used as the second image, the abnormal area is used as the target object, and the image processing model is obtained by training according to the method described in any one of claims 1 to 12; An image and a position of the abnormal area are provided, the first medical image is used as a reference image, and the second medical image is used as an image to be calibrated.
24. A method for processing an image, characterized in that: include: Acquire a first monitoring image and a second monitoring image, wherein the first monitoring image and the second monitoring image are from different monitoring devices, the first monitoring image is used as a reference image, the second monitoring image is used as an image to be calibrated, and the first monitoring image and the second monitoring image are different images of the same monitoring target object; The first monitoring image and the second monitoring image are input into the image processing model to obtain the monitoring target object, and the obtained monitoring target object has the image features of the first monitoring image and the second monitoring image; wherein, in the model training stage, the image processing model is used as the model to be trained, the first monitoring image is used as the first image, the second monitoring image is used as the second image, and the monitoring target object is used as the target object, and the image processing model is obtained by training according to the method described in any one of claims 1 to 12; Identify the information to be found in the monitoring target object.
25. A video processing method, characterized in that: include: Acquire a first surveillance video and a second surveillance video, where the first surveillance video and the second surveillance video are from different surveillance devices; Acquire a first surveillance image from the first surveillance video, and acquire a second surveillance image corresponding to the first surveillance video from the second surveillance video, wherein the first surveillance image is used as a reference image, and the second surveillance image is used as an image to be calibrated, and the acquired first surveillance image and second surveillance image are different images of the same surveillance target object; Input the first monitoring image and the second monitoring image into the image processing model to obtain the monitoring target object, wherein the monitoring target object has the image features of the first monitoring image and the second monitoring image; wherein, in the model training stage, the image processing model is used as the model to be trained, the first monitoring image is used as the first image, the second monitoring image is used as the second image, and the monitoring target object is used as the target object, and the image processing model is obtained by training according to the method described in any one of claims 1 to 12; Identify the information to be found in the monitoring target object.
26. A method for processing an image, characterized in that: include: Acquire a first medical image and a second medical image, wherein the first medical image and the second medical image are images of different modalities for the same abnormal area, the first medical image is used as a reference image, and the second medical image is used as an image to be calibrated; Inputting the first medical image and the second medical image into the model to be trained, obtaining position difference data between the first medical image and the second medical image and the abnormal region, wherein the abnormal region obtained has image features of the first medical image and the second medical image; The model to be trained is trained according to the abnormal area and position difference data.
27. A method for processing an image, characterized in that: include: Acquire a first medical image and a second medical image, wherein the first medical image and the second medical image are different modality images of the same abnormal area; Inputting the first medical image and the second medical image into the image processing model, obtaining an abnormal region in the spliced first medical image and the second medical image, wherein the acquired abnormal region has image features of the first medical image and the second medical image; wherein, in the model training stage, the image processing model is used as a model to be trained, and the image processing model is obtained by training according to the method of claim 26; Obtaining model data of digital virtual human; The model data is fused with the abnormal area so that the fused digital virtual human has the abnormal area.
28. The method according to claim 27, characterized in that The method further comprises: Monitor changes in abnormal areas in the fused digital virtual human.
29. The method according to claim 28, characterized in that The method further comprises: Obtaining the change of the abnormal area through the image processing model; Verify the acquired changes based on the monitored changes.
30. A computing device comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to: Acquire a first image and a second image, wherein the first image is used as a reference image, and the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object; Inputting the first image and the second image into the model to be trained, obtaining position difference data between the first image and the second image and the target object, wherein the obtained target object has image features of the first image and the second image; The model to be trained is trained according to the target object and the position difference data.
31. A computing device comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to: Acquire a first image and a second image, wherein the first image is used as a reference image, and the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object; The first image and the second image are input into an image processing model to obtain the target object, wherein the obtained target object has image features of the first image and the second image, wherein: In the model training stage, the image processing model is used as a model to be trained, and the image processing model is obtained by training according to the method described in any one of claims 1 to 12.
32. A computing device comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to: Acquire a first monitoring image and a second monitoring image, wherein the first monitoring image and the second monitoring image are from different monitoring devices, the first monitoring image is used as a reference image, the second monitoring image is used as an image to be calibrated, and the first monitoring image and the second monitoring image are different images of the same monitoring target object; The first monitoring image and the second monitoring image are input into the image processing model to obtain the monitoring target object, wherein the obtained monitoring target object has the image features of the first monitoring image and the second monitoring image; wherein, In the model training stage, the image processing model is used as the model to be trained, the first monitoring image is used as the first image, the second monitoring image is used as the second image, and the monitoring target object is used as the target object, and the image processing model is obtained by training according to the method described in any one of claims 1 to 12; Identify the information to be found in the monitoring target object.
33. A computing device comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to: Acquire a first surveillance video and a second surveillance video, where the first surveillance video and the second surveillance video are from different surveillance devices; Acquire a first surveillance image from the first surveillance video, and acquire a second surveillance image corresponding to the first surveillance video from the second surveillance video, wherein the first surveillance image is used as a reference image, and the second surveillance image is used as an image to be calibrated, and the first surveillance image and the second surveillance image are different images of the same surveillance target object; The first monitoring image and the second monitoring image are input into the image processing model to obtain the monitoring target object, and the obtained monitoring target object has the image features of the first monitoring image and the second monitoring image; wherein, in the model training stage, the image processing model is used as the model to be trained, the first monitoring image is used as the first image, the second monitoring image is used as the second image, and the monitoring target object is used as the target object, and the image processing model is obtained by training according to the method described in any one of claims 1 to 12; Identify the information to be found in the monitoring target object.
34. A computing device comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to: Acquire a first image and a second image, wherein the first image is used as a reference image, and the second image is used as an image to be calibrated, and the first image and the second image are different images of the same target object; Inputting the first image and the second image into the model to be trained, obtaining position difference data between the first image and the second image through the first sub-model in the model to be trained, obtaining the target object through the position difference data and the second sub-model in the model to be trained, the obtained target object having image features of the first image and the second image, and the training parameters of the first sub-model and the second sub-model are the same; The training parameters are trained according to the target object and the position difference data, thereby training the model to be trained.
35. A computing device comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to: Acquire a first medical image and a second medical image, wherein the first medical image and the second medical image are different modality images of the same abnormal area; The first medical image and the second medical image are input into the image processing model to obtain an abnormal region in the spliced first medical image and the second medical image, wherein the abnormal region has image features of the first medical image and the second medical image; wherein In the model training stage, the image processing model is used as the model to be trained, the first medical image is used as the first image, the second medical image is used as the second image, and the abnormal area is used as the target object, and the image processing model is obtained by training according to the method of claim 26; An image of the abnormal area is provided, the first medical image is used as a reference image, and the second medical image is used as an image to be calibrated.
36. A computing device comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to: Acquire a first medical image and a second medical image, wherein the first medical image and the second medical image are different modality images of the same abnormal area; The first medical image and the second medical image are input into the image processing model, an abnormal region in the spliced first medical image and the second medical image is obtained, and the location of the abnormal region is determined, and the acquired abnormal region has image features of the first medical image and the second medical image; wherein, In the model training stage, the image processing model is used as the model to be trained, the first medical image is used as the first image, the second medical image is used as the second image, and the abnormal area is used as the target object, and the image processing model is obtained by training according to the method of claim 26; An image and a position of the abnormal area are provided, the first medical image is used as a reference image, and the second medical image is used as an image to be calibrated.
37. A computing device comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to: acquire a first medical image and a second medical image, wherein the first medical image and the second medical image are images of different modalities for the same abnormal area, the first medical image is used as a reference image, and the second medical image is used as an image to be calibrated; Inputting the first medical image and the second medical image into the model to be trained, obtaining position difference data between the first medical image and the second medical image and the abnormal region, wherein the abnormal region obtained has image features of the first medical image and the second medical image; The model to be trained is trained according to the abnormal area and position difference data.
38. A computing device comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to: Acquire a first medical image and a second medical image, wherein the first medical image and the second medical image are different modality images of the same abnormal area; The first medical image and the second medical image are input into the image processing model to obtain an abnormal region in the spliced first medical image and the second medical image, wherein the abnormal region has image features of the first medical image and the second medical image; wherein In the model training stage, the image processing model is used as the model to be trained, the first medical image is used as the first image, the second medical image is used as the second image, and the abnormal area is used as the target object, and the image processing model is obtained by training according to the method of claim 26; Obtaining model data of digital virtual human; The model data is fused with the abnormal area so that the fused digital virtual human has the abnormal area.
39. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by one or more processors, the one or more processors are caused to implement the steps in the method according to any one of claims 1 to 29.
Citation Information
Patent Citations
Three-dimensional renal cortex positioning method based on three-dimensional active contour model and three-dimensional Harvard transformation
CN103679805A
Image description generation method and device, model training method and device and storage media
CN108305296A
Image processing method, image processing device and photographing equipment
CN110035218A
Medical image processing method, system and device and computer readable medium
CN110151210A
Breast ultrasound image focus segmentation method based on semi-supervised learning
CN110189323A
