Entity detection methods and related devices

Abstract

The application relates to an entity detection method and related equipment, which comprises the following steps: acquiring a first image and a second image; detecting an entity change region of the second image relative to the first image to obtain change region information; extracting a first region image from the first image according to the change region information, and extracting a second region image from the second image; performing entity segmentation on the first region image to obtain a first entity pixel region in which at least one entity in the first region image is located; performing entity segmentation on the second region image to obtain a second entity pixel region in which at least one entity in the second region image is located; and performing entity matching according to the first entity pixel region and the second entity pixel region to determine a change attribute of the entity between the first image and the second image. The application guarantees the accuracy of the determined change attribute, and can be applied to scenes including but not limited to a map, geographic information and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and more particularly, to an entity detection method and related equipment. BACKGROUND

[0002] Entities (e.g., buildings) in a geographical environment can change, for example, a certain building is demolished, rebuilt, or a new building is built in an area where there was no building originally. For an electronic map product, the model of the entity shown in the electronic map needs to be consistent with the entity in the environment, therefore, the map needs to be updated in real time according to the changes of the entity in the geographical environment. However, how to determine the change attribute of the entity is a technical problem to be solved in the prior art. SUMMARY

[0003] In view of the above problems, the embodiments of the present application propose an entity detection method and related equipment to automatically detect and determine the change attribute of an entity.

[0004] According to an aspect of an embodiment of the present application, an entity detection method is provided, which includes: acquiring a first image and a second image, the acquisition time corresponding to the first image being different from the acquisition time corresponding to the second image; detecting an entity change area of the second image relative to the first image to obtain change area information; performing region extraction from the first image to obtain a first region image according to the change area information, and performing region extraction from the second image to obtain a second region image; performing entity segmentation on the first region image to obtain a first entity pixel region where at least one entity in the first region image is located; performing entity segmentation on the second region image to obtain a second entity pixel region where at least one entity in the second region image is located; and performing entity matching according to the first entity pixel region and the second entity pixel region to determine a change attribute of the at least one entity between the first image and the second image.

[0005] According to an aspect of some embodiments of the present application, an entity detection apparatus is provided, comprising: an image acquisition module configured to acquire a first image and a second image, wherein the first image corresponds to a first acquisition time and the second image corresponds to a second acquisition time; a change region detection module configured to detect an entity change region of the second image relative to the first image to obtain change region information; a region extraction module configured to extract a first region image from the first image and a second region image from the second image according to the change region information; a first entity segmentation module configured to perform entity segmentation on the first region image to obtain a first entity pixel region in which at least one entity in the first region image is located; a second entity segmentation module configured to perform entity segmentation on the second region image to obtain a second entity pixel region in which at least one entity in the second region image is located; and a change attribute determination module configured to perform entity matching according to the first entity pixel region and the second entity pixel region to determine a change attribute of the at least one entity between the first image and the second image.

[0006] In some embodiments of the present application, based on the foregoing scheme, the change region detection module comprises: a difference unit configured to perform difference between the first image and the second image to obtain a difference image; a first feature down-sampling unit configured to perform feature down-sampling on the difference image step by step to obtain at least two levels of first down-sampling feature maps; a feature pyramid fusion unit configured to perform feature pyramid fusion on the at least two levels of first down-sampling feature maps to obtain feature maps at at least two scales; a change region prediction unit configured to perform entity change region detection on the feature maps at the at least two scales respectively to obtain at least two groups of reference change region information; and a non-maximum suppression unit configured to perform non-maximum suppression according to the change regions respectively indicated by the at least two groups of reference change region information to obtain the change region information.

[0007] In some embodiments of the present application, based on the foregoing scheme, the feature maps at the at least two scales include a first feature map, at least one second feature map, and at least one third feature map; the feature pyramid fusion unit includes: a first feature map determination unit configured to determine, as the first feature map, a down-sampling feature map with the largest number of down-sampling levels among the at least two levels of down-sampling feature maps; a convolution processing unit configured to perform convolution processing on the first feature map level by level to obtain the at least one second feature map; a first feature up-sampling unit configured to perform up-sampling on the first feature map to obtain a first up-sampled feature map; and a fusion unit configured to fuse the first up-sampled feature map and a first down-sampled feature map with the same size as the first up-sampled feature map to obtain an initial third feature map and update a fusion frequency; if the fusion frequency does not reach a fusion frequency threshold, the down-sampling and fusion are continued based on the initial third feature map, and the fusion frequency is updated until the fusion frequency reaches the fusion frequency threshold.

[0008] In some embodiments of the present application, based on the foregoing scheme, the second entity segmentation module includes: a second feature down-sampling unit configured to perform feature down-sampling on the second region image level by level to obtain shared features of the second region image; an entity feature extraction unit configured to perform entity feature extraction based on the shared features of the second region image to obtain entity features of each pixel in the second region image; a semantic feature extraction unit configured to perform semantic feature extraction based on the shared features of the second region image to obtain semantic features of each pixel in the second region image, the semantic features being used to indicate whether the corresponding pixel is foreground; and a clustering unit configured to perform clustering based on the entity features of each pixel in the second region image and the semantic features of each pixel in the second region image to determine second entity pixel regions in which each entity in the second region image is located.

[0009] In some embodiments of the present application, based on the foregoing scheme, the entity feature extraction unit includes: a second feature up-sampling unit configured to perform feature up-sampling on the shared features of the second region image level by level to obtain first up-sampled features of the second region image; and a pixel embedding unit configured to perform pixel embedding based on the first up-sampled features of the second region image to obtain the entity features of each pixel in the second region image.

[0010] In some embodiments of the present application, based on the foregoing scheme, the semantic feature extraction unit includes: a third feature up-sampling unit configured to perform feature up-sampling on the shared features of the second region image level by level to obtain second up-sampled features of the second region image; and a binary segmentation unit configured to perform binary segmentation based on the second up-sampled features of the second region image to obtain the semantic features of each pixel in the second region image.

[0011] In some embodiments of the present application, based on the foregoing scheme, the entity detection apparatus further comprises a spatial position information determination module configured to predict spatial position information of the entity in the second region image according to the shared features of the second region image.

[0012] In some embodiments of the present application, based on the foregoing scheme, the spatial position information comprises entity offset information; and the spatial position information determination module comprises: a fourth feature up-sampling unit configured to perform feature up-sampling on the shared features of the second region image step by step to obtain third up-sampled features of the second region image; an offset position prediction unit configured to perform offset position prediction according to the third up-sampled features of the second region image to obtain position offset information corresponding to each pixel in the second region image; and an entity offset information determination unit configured to determine entity offset information of the entity indicated by the second entity pixel region in the second region image according to the position offset information corresponding to each pixel in the second entity pixel region in the second region image.

[0013] In some embodiments of the present application, based on the foregoing scheme, the spatial position information comprises entity height; and the spatial position information determination module comprises: a fifth feature up-sampling unit configured to perform feature up-sampling on the shared features of the second region image step by step to obtain fourth up-sampled features of the second region image; a height information prediction unit configured to predict height information corresponding to each pixel in the second region image according to the fourth up-sampled features of the second region image; and an entity height determination unit configured to determine entity height of the entity indicated by the second entity pixel region in the second region image according to the height information corresponding to each pixel in the second entity pixel region in the second region image.

[0014] In some embodiments of the present application, based on the foregoing scheme, the entity height determination unit comprises: a height median value determination unit configured to determine a height median value of the second entity pixel region according to the height information corresponding to each pixel in the second entity pixel region; and a determination unit configured to determine the height median value of the second entity pixel region as the entity height of the entity indicated by the second entity pixel region.

[0015] In some embodiments of the present application, based on the foregoing scheme, the entity detection apparatus further comprises an updating module configured to update model data of the entity in the map according to the spatial position information of the entity in the second region image and the change attribute of the entity in the second region image.

[0016] In some embodiments of the present application, based on the foregoing scheme, the change attribute determination module comprises: an intersection-over-union calculation unit, configured to determine an intersection-over-union of the first entity pixel region and the second entity pixel region according to the first entity pixel region and the second entity pixel region; and a change attribute determination unit, configured to determine a change attribute corresponding to the intersection-over-union as the change attribute of the corresponding entity between the first image and the second image.

[0017] According to an aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; a memory, having computer readable instructions stored thereon, which, when executed by the processor, implement the entity detection method as described above.

[0018] According to an aspect of an embodiment of the present application, a computer readable storage medium is provided, having computer readable instructions stored thereon, which, when executed by a processor, implement the entity detection method as described above.

[0019] In the scheme of the present application, first, the entity change region is detected according to the first image and the second image, and then the region extraction is performed from the first image and the second image according to the obtained change region information, the first region image and the second region image extracted from the first image are subjected to entity segmentation, and then the entity pixel region of each entity in the first region image and the entity pixel region of each entity in the second region image are subjected to entity matching to determine the change attribute of the entity, thereby realizing the automatic detection and determination of the change attribute of the entity in stages.

[0020] Moreover, in the entity change region detection stage, the first image and the second image are taken as the granularity; in the entity segmentation stage, the extracted first region image and the second region image are taken as the granularity; and in the entity matching stage, the first entity pixel region and the second entity pixel region where the entity is located are taken as the granularity, so that the detection granularity gradually decreases, thereby effectively ensuring the accuracy of the determined change attribute of the entity. In the entity change region detection stage, the first image and the second image with a larger granularity are detected, which can effectively ensure the recall rate of the detection.

[0021] Further, if the environment conditions (such as light, weather, and seasonal change) at the time of acquisition of the first image and the second image are different, resulting in false detection in the entity change region detection stage, in the present scheme, since the entity change region is determined, the entity segmentation and entity matching are performed at a smaller granularity, which can effectively identify the region that is incorrectly determined as the entity change region, thereby effectively ensuring the accuracy of the determined result of the change attribute of the entity. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application. It is to be understood that the drawings are designed solely for purposes of illustration to be used in conjunction with the description insofar as specifically described herein. Changes in design or structure can be made in the drawings without corresponding to the specific description herein.

[0023] Figure 1 is a schematic diagram of an implementation environment according to an embodiment of the present application.

[0024] Figure 2 is a flowchart of an entity detection method according to an embodiment of the present application.

[0025] Figure 3 is a schematic diagram of the intersection and union of two pixel regions according to an embodiment of the present application.

[0026] Figure 4 is a flowchart of step 220 according to an embodiment of the present application.

[0027] Figure 5 is an architecture diagram of a target detection network of an FCOS algorithm according to an embodiment of the present application.

[0028] Figure 6 is a flowchart of step 250 according to an embodiment of the present application.

[0029] Figure 7 is a flowchart of predicting entity offset information according to an embodiment of the present application.

[0030] Figure 8 is a flowchart of predicting entity height according to an embodiment of the present application.

[0031] Figure 9 is an architecture diagram of an entity segmentation network according to an embodiment of the present application.

[0032] Figure 10 is a flowchart of determining a change attribute of an entity according to an embodiment of the present application.

[0033] Figures 11-1 to 11-4 is a schematic diagram of a determination result of a change attribute of a building according to an embodiment of the present application.

[0034] Figure 12 is a block diagram of an entity detection apparatus according to an embodiment of the present application.

[0035] Figure 13 shows a structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0036] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations can be implemented in any

[0037] Moreover, described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the

[0038] The block diagrams in the drawings show only the functionality of the embodiments and do not imply any particular physical or architectural arrangement of the devices, systems, or methods. No inference should be drawn regarding the implementational aspects of the embodiments as shown and described herein. Further, having described a functionality of several embodiments, additional embodiments having different functionalities can also be implemented.

[0039] The flow diagrams depicted herein are examples of sequences of operations that can be performed, for example, by a computing device. The depicted examples are not meant to be limiting, as one of skill in the art could readily devise many other variations of the sequences of operations that would be within the scope of the present application. For example, the order of some of the operations can be changed, some or all of the operations can be changed to other operations, and some of the operations can be combined, or some of the operations can be broken into multiple operations.

[0040] It should be noted that "a plurality" refers to two or more. The conjunction "and / or" describes associative relationships of associated objects, which means that there can be three relationships, for example, A and / or B can represent three cases: A alone, A and B together, and B alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0041] In the related art, entities (e.g., buildings) in a geographic environment can change, for example, a certain building is demolished, rebuilt, or a new building is built in an area where there was no building originally, and for a map product, it is necessary to update the map in real time according to the changes of the entities in the geographic environment. Therefore, it is necessary to determine the change attribute of the entity in the geographic environment according to the image. In the scheme of the present application, a method for automatically detecting and determining the change attribute of the entity based on computer vision technology is provided.

[0042] Computer Vision (CV) is the application of artificial intelligence technology on images. Computer vision is a science that studies how to make machines "see", that is, to replace human eyes with image acquisition devices and computer devices for target identification, tracking and measurement, and further image processing, such as image recognition, image semantic understanding, image retrieval, three-dimensional object reconstruction, simultaneous localization and mapping, and other technologies.

[0043] Artificial Intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0044] Figure 1 is a schematic diagram of an implementation environment according to an embodiment of the present application. As shown in Figure 1 , the implementation environment includes a terminal 110 and a server 120 communicatively connected with the terminal 110. The terminal 110 can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a self-service terminal, a vehicle-mounted terminal, a smart television, etc., which is not specifically limited here. The server 120 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0045] The terminal 110 can select images to select first and second images with different acquisition times, obtain an image pair including the first and second images, and then send the image pair to the server 120, so that the server 120 determines the change attribute of the entity between the first and second images according to the method of the present application, and obtains a change attribute determination result. The server 120 can further send the change attribute determination result to the terminal 110.

[0046] In some embodiments of the present application, after the server 120 determines the change attribute of the entity in the image, the change attribute of the entity in the first and / or second image can be further marked, so as to obtain a marked image with the change attribute mark, and the marked image is sent to the terminal 110, so that the user knows the change attribute of each entity.

[0047] In some embodiments of the present application, the first image and the second image can be satellite images. In this scenario, the terminal 110 can further display a geographical area selection page, in which a city can be selected, and a sub-region in the city can be selected, for example, a city and a street region in the city can be selected, so that satellite images collected at two time points for a target city and a target region selected by a user are acquired, and then the server 120 determines the change attribute of an entity in the two satellite images according to the method of the present application. Further, the server can update the model data of the entity in the map based on the determined change attribute of the entity.

[0048] It should be noted that the functions of the terminal 110 can be integrated in the server 120, or the functions of the server 200 can be integrated in the terminal 110, in which case the method of the present application is implemented on one device (terminal or server).

[0049] The implementation details of the technical solutions of the embodiments of the present application are described in detail as follows:

[0050] Figure 2 is a flowchart of an entity detection method according to an embodiment of the present application, which can be executed by a computer device with processing capability, for example, a server in Figure 1 , and the like, which is not specifically limited here. Referring to Figure 2 , the method includes at least steps 210 to 260, which are described in detail as follows:

[0051] Step 210, acquiring a first image and a second image, the acquisition time corresponding to the first image being different from the acquisition time corresponding to the second image.

[0052] In some embodiments of the present application, the first image or the second image can be an image directly collected by an image collection device, or can be an image obtained by preprocessing an image directly collected by an image collection device, which is not specifically limited here.

[0053] If the first image or the second image is an image obtained by preprocessing an image directly collected by an image collection device, the acquisition time corresponding to the first image or the second image is the acquisition time of the original image from which the first image or the second image is derived.

[0054] In some embodiments of the present application, in order to ensure the efficiency of image processing, the first image and the second image can be images of the same geographical region, so that the change attribute of the determined entity is more targeted.

[0055] In some embodiments of the present application, to avoid the influence of other factors on the determined changed attribute, the first image and the second image are images of the same geographical area at different time points under the same perspective. Specifically, the first image and the second image can be images captured by an image capturing device at the same or similar position at different times according to the same image capturing parameters (e.g. focal length), so that the images presented in the first image and the second image are images of the same geographical area under the same perspective.

[0056] Of course, in other embodiments, to ensure that the first image and the second image present images of the same geographical area under the same perspective, images captured by image capturing devices at different positions, under different perspectives, and with different focal lengths at different time points can also be pre-processed, such as image enlargement or reduction, image segmentation, perspective transformation, etc., to ensure that the first image and the second image obtained after pre-processing have the same size and present the same geographical area under the same perspective. It is worth mentioning that the same referred to above can mean exactly the same, or that the difference between the two is within a set difference range, so that the two are basically the same.

[0057] Step 220, detecting the entity changed area of the second image relative to the first image, and obtaining changed area information.

[0058] With the passage of time, entities in the same geographical area can change, so the entities presented by the first image and the second image corresponding to different capture times can be different, i.e. the entities presented in the second image can change relative to the entities in the first image. The area in the image including the changed entities is referred to as the entity changed area. The obtained changed area information is used to indicate the position of the entity changed area in the first image / second image.

[0059] Entities, which can also be referred to as instances, refer to physical objects in a geographical area, such as buildings, vehicles, tracks, lakes, forests, grasslands, etc., which are not specifically limited here. Entities change, such as being rebuilt, newly built, demolished, etc. It can be understood that which change an entity undergoes is related to the image selected as a reference, for example, assuming that the capture time corresponding to the first image is earlier than that of the second image, the first image can be taken as a reference image and the second image as a contrast image, if an entity exists in the second image but not in the first image, it indicates that the entity is a newly built entity; if an entity does not exist in the second image but exists in the first image, it indicates that the entity is a demolished entity; if an entity exists in both the first image and the second image, but the pixel area occupied in the first image is significantly different from that in the second image, it indicates that the entity is a rebuilt entity.

[0060] If there is an entity that changes in the two images, it is directly reflected in the difference between the first image and the second image in pixels. Therefore, in some embodiments of the present application, the first image and the second image can be differentiated to obtain a difference image, and then the entity change region can be determined according to the difference image.

[0061] In some embodiments of the present application, in order to facilitate the differentiation of the first image and the second image, the sizes of the first image and the second image are set to be the same, and the number of pixels in the length and width directions of the first image and the second image is the same, so that when the image is differentiated, any pixel can be taken as a starting point for pixel differentiation.

[0062] In some embodiments of the present application, a target detection algorithm can be used to detect the entity change region in the difference image, that is, the region where the entity changes is taken as a target, and the position information of the entity change region in the difference image is detected according to the target detection algorithm. The target detection algorithm can be an FCOS (Fully Convolutional One-Stage Object Detection) algorithm, a Faster R-CNN (Faster Region-Convolutional Neural Network) algorithm, etc., which is not specifically limited here.

[0063] It can be understood that there can be multiple regions where the entity changes in the first image and the second image, and correspondingly, the obtained change region information can be multiple groups, and one group of change region information is used to indicate the position of one entity change region.

[0064] Step 230, according to the change region information, region extraction is performed on the first image to obtain a first region image, and region extraction is performed on the second image to obtain a second region image.

[0065] As described above, the change region information indicates the position of the detected and determined entity change region. Therefore, according to the change region information, the image region indicated by the change region information can be extracted from the first image, that is, the first region image; and the image region indicated by the change region information can be extracted from the second image, that is, the second region image.

[0066] Step 240, entity segmentation is performed on the first region image to obtain a first entity pixel region where at least one entity in the first region image is located.

[0067] Step 250, entity segmentation is performed on the second region image to obtain a second entity pixel region where at least one entity in the second region image is located.

[0068] Entity segmentation, which can also be referred to as instance segmentation, refers to distinguishing different entities of the same class in an image on the basis of semantic segmentation of the image. For example, if the image is an image including multiple buildings, the pixel regions where the multiple buildings are located need to be distinguished, and the position of each building in the image needs to be determined. Semantic segmentation refers to predicting and determining which class each pixel in the image belongs to. For example, if only the buildings in the image need to be focused on, the type labels that are set can include a label indicating that the pixel is a building and a label indicating that the pixel is not in the region of the building.

[0069] In some embodiments of the present application, entity segmentation can be performed by Mask R-CNN (Mask Recycle Convolutional Neural Network), PANet (Path Aggregation Network), and FCIS (Fully Convolutional Instance-aware Semantic Segmentation) network, which are not specifically limited herein.

[0070] In the scheme of the present application, for the sake of distinction, the entity pixel region where the determined entity in the first region image is located is referred to as the first entity pixel region, and the entity pixel region where the determined entity in the second region image is located is referred to as the second entity pixel region.

[0071] In steps 240 and 250, entity segmentation is performed to determine the entity pixel region (i.e., the first entity pixel region) where each entity in the first region image is located and the entity pixel region (i.e., the second entity pixel region) where each entity in the second region image is located.

[0072] In step 260, entity matching is performed according to the first entity pixel region and the second entity pixel region to determine the change attribute of the entity between the first image and the second image.

[0073] Entity matching refers to determining whether the entity indicated by the first entity pixel region and the entity indicated by the second entity pixel region are the same entity according to the overlap (intersection) of the first entity pixel region and the second entity pixel region indicating the same geographical location.

[0074] In some embodiments of the present application, the change attribute of the entity can include no change, reconstruction, new construction, and demolition. Corresponding to each change attribute, the intersection region between the pixel region occupied by the entity in the first image and the pixel region occupied by the entity in the second image is different, and therefore, the change attribute of the entity can be determined according to the intersection region between the first entity pixel region and the second entity pixel region.

[0075] In some embodiments of the present application, a corresponding relationship between the pixel area range and the change attribute is set, so that after the pixel area of the intersection region is determined, the change attribute corresponding to the pixel area range to which the pixel area of the intersection region belongs is determined as the change attribute of the entity.

[0076] In some embodiments of the present application, in order to avoid the influence of the difference of the base pixel area of the entity on the determined change attribute, the ratio of the pixel area of the intersection region to the reference pixel area can also be used as a target parameter, the target parameter range corresponding to each change attribute is set in advance, and then after the pixel area of the intersection region is determined, the target parameter is calculated correspondingly, and the change attribute corresponding to the target parameter range to which the calculated target parameter belongs is determined as the change attribute of the entity. In specific embodiments, the reference pixel area can be the pixel area of the first entity pixel region, can also be the pixel area of the second entity pixel region, can also be the average pixel area of the first entity pixel region and the second entity pixel region, and can also be the union pixel area of the first entity pixel region and the second entity pixel region, which can be set according to actual needs, and is not limited here.

[0077] In some embodiments of the present application, step 260 comprises: determining the intersection over union (IOU) of the first entity pixel region and the second entity pixel region according to the first entity pixel region and the second entity pixel region; and determining the change attribute corresponding to the intersection over union as the change attribute of the entity between the first image and the second image.

[0078] The intersection over union (IOU) of the first entity pixel region and the second entity pixel region is equal to the ratio of the intersection to the union of the first entity pixel region and the second entity pixel region. Specifically, the pixel area of the intersection of the first entity pixel region and the second entity pixel region and the pixel area of the union are calculated first, and then the pixel area of the intersection is divided by the pixel area of the union to obtain the intersection over union.

[0079] The intersection over union of the two can well judge their positional relationship and overlapping degree. When the intersection over union of the two is large, it is considered that they are highly overlapped, and in this case, it is determined that the entities represented by the first entity pixel region and the second entity pixel region are the same, that is, the entity does not change in the first image and the second image.

[0080] Figure 3 is a schematic diagram for determining the intersection and union of two pixel regions according to an embodiment of the present application. As shown in Figure 3As shown, assuming that the rectangular region X is the first entity pixel region and the rectangular region Y is the second entity pixel region, the intersection of the rectangular region X and the rectangular region Y is the intersection of the first entity pixel region and the second entity pixel region, and the total region covered by the rectangular region X and the rectangular region Y is the union of the first entity pixel region and the second entity pixel region. On this basis, the intersection-union ratio is obtained by dividing the pixel area of the intersection by the pixel area of the union.

[0081] In some embodiments of the present application, the intersection-union ratio range corresponding to each change attribute can be preset, so that after the intersection-union ratio of the first entity pixel region and the second entity pixel region is calculated, the change attribute corresponding to the intersection-union ratio range to which the calculated intersection-union ratio belongs is determined as the change attribute of the entity.

[0082] In some embodiments of the present application, after the change attribute of each entity in the first region image and the second region image is determined, entity markers (for example, rectangular frames) can be added to the entities in the first region image and the second region image in the first image and the second image, and region markers (for example, rectangular frames) can be added to the first region image and the second region image, and the marked first image and the second image are returned to the terminal for the user to view the change attribute determination result of the entity. In some embodiments of the present application, different styles of region markers can be added to the first region image and the second region image according to the change attribute, and the different styles of region markers can be different colors and different line types of the region markers, which are not limited here.

[0083] In the scheme of the present application, the change region is first detected according to the first image and the second image, and the region extraction is performed from the first image and the second image according to the obtained change region information, the first region image and the second region image extracted from the first image are subjected to entity segmentation, and the entity pixel region of each entity in the first region image and the entity pixel region of each entity in the second region image are subjected to entity matching to determine the change attribute of the entity, which realizes the automatic determination of the change attribute of the entity in stages.

[0084] Moreover, in the entity change region detection stage, the detection is performed with the first image and the second image as the granularity, in the entity segmentation stage, the entity segmentation is performed with the extracted first region image and the second region image as the granularity, and in the entity matching stage, the entity matching is performed with the first entity pixel region and the second entity pixel region where the entity is located as the granularity, so that the detection granularity gradually decreases, which effectively ensures the accuracy of the determined change attribute of the entity. Moreover, in the entity change region detection stage, the detection is performed with the first image and the second image as the granularity, which can effectively ensure the recall rate of the entity change region detection.

[0085] Further, due to the different environmental conditions (e.g. light, weather, seasonal changes) when the first image and the second image are captured, there may be false detection in the entity change area detection stage. For example, in winter, the roof is covered with thick snow, and the color change is large, which will be detected as an entity change area. Since the entity change area is detected, the entity segmentation and entity matching are performed at a smaller granularity, which can effectively identify the area that is incorrectly determined as an entity change area, and effectively ensure the accuracy of the entity change attribute determination result.

[0086] In some embodiments of the present application, as shown in Figure 4 Step 220 includes step 410, which differentiates the first image and the second image to obtain a difference image. The pixels in the first image are differentiated from the pixels in the second image, and a difference image is obtained. The differentiation can be performed on at least one of the gray value of the pixels or the color content on each color channel.

[0087] For the first image and the second image of the same geographical area at different time points, most of the entities in the first image and the second image are the same, and the pixels of the same entities in the image are basically the same. The difference image obtained by differentiation filters out the same or less different pixels in the first image and the second image, and retains the information of the pixels with differences, so that in the subsequent processing process, the pixels with differences can be mainly focused on and the pixels without differences or with less differences can be ignored.

[0088] Step 420 performs feature down-sampling on the difference image level by level to obtain at least two levels of first down-sampled feature maps. In some embodiments of the present application, a neural network, such as a convolutional neural network, can be used to extract features from the difference image. In the process of extracting features from the difference image, a multi-layer neural network can be used to extract features layer by layer. Each layer of the neural network performs feature down-sampling based on the down-sampled feature map output by the previous layer of the neural network, and outputs the down-sampled feature map of the difference image on the current layer of the neural network. The feature map that serves as the basis for feature down-sampling is referred to as an initial feature map.

[0089] In the process of performing feature down-sampling level by level, the down-sampled feature map obtained by the current level of feature down-sampling is output as a feature map, and is input to the next level of feature down-sampling as an input, so as to obtain at least two levels of down-sampled feature maps. In specific embodiments, the number of levels of feature down-sampling can be set according to actual needs, and is not specifically limited here. Figure 1

[0090] In some embodiments of the present application, a plurality of cascaded residual networks can be used to perform feature down-sampling on the initial feature map level by level. For each level of residual network, the input of the current level of residual network is the output of the previous level of residual network.​

[0091] In the downsampling process, the size of the feature map is reduced, so that the receptive field of the convolution can be increased, and multi-level downsampling can facilitate the subsequent fusion of multi-scale features.

[0092] At step 430, feature pyramid fusion is performed according to the at least two levels of first down-sampled feature maps to obtain feature maps at at least two scales.

[0093] For the obtained at least two levels of down-sampled feature maps, the down-sampled feature map with a smaller number of down-sampling levels has a higher resolution and contains more position and detail information, but the number of convolution processes it undergoes is small, so it has less semantic information and more noise. The down-sampled feature map with a larger number of down-sampling levels has a larger number of convolution processes, so it has more semantic information, but it has a low resolution and poor perception of details.

[0094] Feature pyramid fusion (Feature Pyramid Networks, FPN) refers to upsampling the down-sampled feature map with a large number of down-sampling levels and adding it element by element to the down-sampled feature map with a small number of down-sampling levels, thereby obtaining feature maps of different scales, and connecting the down-sampled feature map with a small number of down-sampling levels (shallow feature map) and the down-sampled feature map with a large number of down-sampling levels (deep feature map) to pass the information of the shallow layer to the deep layer, so as to solve the problem that the deep feature map easily ignores details.

[0095] In some embodiments of the present application, the feature maps at at least two scales include a first feature map, at least one second feature map, and at least one third feature map. In this embodiment, step 440 includes: taking the down-sampled feature map with the largest number of corresponding down-sampling levels in the at least two levels of down-sampled feature maps as the first feature map; performing convolution processing on the first feature map level by level to obtain at least one second feature map; upsampling the first feature map to obtain a first up-sampled feature map; fusing the first up-sampled feature map and the first down-sampled feature map with the same size as the first up-sampled feature map to obtain an initial third feature map and updating the fusion times; if the fusion times have not reached the fusion times threshold, continuing to perform downsampling and fusion on the initial third feature map, i.e., downsampling the initial third feature map and fusing the downsampling result and the first down-sampled feature map with the same size as the downsampling result to obtain a next-level third feature map and update the fusion times correspondingly; repeating the process until the updated fusion times reach the fusion times threshold.

[0096] In the embodiment, the first feature map is further subjected to convolution processing, so that the obtained at least one level of convolution feature map has a larger receptive field than the first feature map, and thus can be used for predicting a larger entity change region. The receptive field of the obtained at least one level of fusion feature map is smaller than that of the at least one level of convolution feature map, and thus can be used for predicting a smaller entity change region.

[0097] In some embodiments of the present application, each up-sampling feature map can be added to the corresponding down-sampling feature map to obtain a corresponding third feature map.

[0098] At step 440, entity change region detection is performed on the feature maps at the at least two scales respectively to obtain at least two groups of reference change region information.

[0099] For each feature map at each scale, change region prediction is performed to obtain a group of predicted change region information, so that according to the feature maps at the at least two scales, at least two groups of reference change region information can be obtained.

[0100] At step 450, according to the change regions indicated by the at least two groups of reference change region information respectively, non-maximum suppression is performed to obtain change region information.

[0101] Non-maximum suppression (NMS) refers to searching for local maximum values and suppressing regions that are not maximum values. The obtained at least two groups of reference change region information indicate at least two entity change regions, and there can be overlaps between the at least two entity change regions. Therefore, the method of non-maximum suppression is used to eliminate the bounding boxes of redundant entity change regions and find the best target bounding box. The region where the determined best target bounding box is located is the entity change region finally determined.

[0102] In the embodiment, the feature pyramid fusion is used to obtain the feature maps at the at least two scales, and the entity change region detection is performed based on the feature maps at the at least two scales. Since the receptive fields of the feature maps at different scales are different, different sizes of entity change regions can be detected according to the feature maps at the at least two scales, so as to ensure the accuracy of the detected entity change regions and the recall rate of the determined entity change regions.

[0103] Figure 5 Fig. 1 is a schematic diagram of a target detection network based on the FCOS algorithm according to some embodiments of the present application, which can be used to detect and determine entity change regions. As shown in Fig. 1, the target detection network includes a basic feature extraction network 510, a feature pyramid fusion network 520, and a detector head 530. Figure 5 As shown in Fig. 1, the target detection network includes a basic feature extraction network 510, a feature pyramid fusion network 520, and a detector head 530.

[0104] After the first image and the second image are differentiated to obtain the difference image, the difference image 511 is input into the base feature extraction network 510 for step-by-step feature down-sampling. In this embodiment, the step-by-step feature down-sampling can be performed by cascading multiple residual units, such as Resnet101 residual units. Each residual unit includes a convolution layer, a normalization layer, an activation layer, and an identity mapping layer. The convolution layer at the bottom is responsible for extracting basic features such as edge textures in the image, and the convolution layer at the higher layer is responsible for combining and abstracting the texture features at the bottom. The normalization layer (Batch Normalization) is used to normalize the features to a normal distribution. The activation layer (Relu) performs a non-linear mapping on the extracted features to enhance the generalization ability of the network. In this embodiment, the difference image is down-sampled for 5 levels, and the first down-sampled feature map C3, C4, and C5 obtained by the 3rd, 4th, and 5th level down-sampling, respectively, are used as the basis for feature fusion. Figure 5 Figure 5 The size of the first down-sampled feature map C3, C4, and C5 is HxW / s, where H represents the height of the image (feature map), W represents the width, and s represents the number of channels.

[0105] The feature pyramid fusion network 520 first takes the first down-sampled feature map C5 with the highest down-sampling level as the first feature map P5, and then performs a convolution operation with a step size of 2 on the first feature map P5 for two consecutive levels to obtain the second feature map P6 and the second feature map P7, respectively. On the basis of the first feature map P5, the feature map obtained by 2 times up-sampling of the first feature map P5 (i.e., the first up-sampled feature map) is added to the down-sampled feature map C4 with the same size as the first up-sampled feature map to obtain the initial third feature map P4; and the feature map obtained by 2 times up-sampling of the initial third feature map P4 (another first up-sampled feature map) is added to the down-sampled feature map C3 with the same size as the first up-sampled feature map to obtain the second-level third feature map P3. The first feature map P5 is up-sampled by 2 times to enlarge the size of the first feature map P5, so that the size of the first up-sampled feature map obtained by the up-sampling is the same as that of the down-sampled feature map C4, facilitating the addition and fusion of the two.

[0106] Since the convolution depth of the second feature map P6 and the second feature map P7 is larger, and the receptive field is larger, the second feature map P6 and the second feature map P7 can be used to detect larger entity change regions. The first feature map P5, the third feature map P3, and the fourth feature map P4 are used to detect smaller entity change regions.

[0107] ​After obtaining the feature maps at multiple scales (i.e., the first feature map P5, the second feature maps P6 and P7, and the third feature maps P3 and P4), the shared detector head 530 predicts the location of the entity change region in the difference image according to the feature map at each scale, respectively.

[0108] As shown in Figure 5 , the detector head includes three branches: a type branch, a center point branch, and a detection branch, wherein the type prediction branch outputs a classification, the center point branch outputs a center point position, and the detection branch outputs a distance from the center point to the upper, lower, left, and right edges. During training of the target detection network, the network is trained according to the loss functions respectively set for the type branch, the center point branch, and the detection branch.

[0109] In the scheme of the present embodiment, an anchor-free strategy is adopted, i.e., the preset anchor box size and scale are eliminated, all related intersection over union calculations and matching are avoided, and hyperparameters are selected, thereby saving memory. Each pixel directly predicts the target (i.e., whether it is an entity change region), measures the distance of the upper, lower, left, and right edges of the target box, and the category of the target, completely avoiding complex calculations related to anchor boxes. In forward inference, the position box of the entity change region is predicted based on the feature maps at different P3-P7, and non-maximum suppression is used to ensure the number of effective boxes.

[0110] In some embodiments of the present application, as shown in Figure 6 , step 250 includes:

[0111] Step 610: performing feature down-sampling on the second region image at multiple levels to obtain shared features of the second region image.

[0112] In some embodiments of the present application, the feature down-sampling on the second region image at multiple levels can be performed by a plurality of cascaded convolutional networks, and the feature map output by the last convolutional network is taken as the shared features of the second region image. In some embodiments of the present application, the feature maps output by the convolutional networks at different levels for the second region image can also be fused, and the fusion result is taken as the shared features of the second region image. In some other embodiments of the present application, the set of feature maps respectively output by the convolutional networks at different levels for the second region image can also be taken as the shared features of the second region image.

[0113] Step 620: performing entity feature extraction according to the shared features of the second region image to obtain entity features of each pixel in the second region image.

[0114] In some embodiments of the present application, step 620 comprises: performing feature upsampling on the shared features of the second region image level by level to obtain first upsampled features of the second region image; and performing pixel embedding on the first upsampled features of the second region image to obtain entity features of each pixel in the second region image.

[0115] The feature upsampling is performed on the basis of the shared features of the second region image to extract deeper level features based on the shared features for subsequent pixel embedding. The first upsampled features of the second region image comprise pixel feature vectors of each pixel in the second region image.

[0116] The pixel embedding refers to a process of mapping a high-dimensional feature vector of a pixel to a low-dimensional feature vector to compress the features of the pixel. Through the pixel embedding, the distance between pixels with higher similarity is smaller in the pixel embedding space, and vice versa, the distance between pixels with lower similarity is farther. The entity feature of a pixel refers to a feature vector of the pixel in the pixel embedding space, and the entity feature of the pixel reflects the classification of the pixel in the pixel region where the entity is located. Through the pixel embedding, on the one hand, the main features of the second region image are retained, and on the other hand, the feature compression is realized to reduce the amount of calculation for subsequent processing.

[0117] For example, if the entity is a building, if the image is collected from the top of the building downwards, since the image collection cannot guarantee that the angle of view of the image collection is completely perpendicular to the base of the building, therefore, the collected image including the building may include not only the top region of the building but also the side region of the building. Through the pixel embedding, the pixel regions of different surfaces of the same building in the image can be distinguished, for example, the pixel region where the top of the building is located and the pixel region where the left side of the building is located are distinguished, that is, the distance between the pixel feature vectors (entity features) of the pixels in the same surface region of the building in the pixel embedding space is smaller, and the distance between the pixel feature vectors of the pixels in different surface regions of the building in the pixel embedding space is larger.

[0118] Step 630 comprises: performing semantic feature extraction on the shared features of the second region image to obtain semantic features of each pixel in the second region image. The semantic feature is used to indicate whether the corresponding pixel is foreground.

[0119] In the scheme of the present application, the foreground refers to the pixel region where the entity is located, in other words, if a pixel is in the pixel region corresponding to the entity, the semantic feature of the pixel indicates that the pixel is foreground; otherwise, if a pixel is not in the pixel region of the entity, the semantic feature of the pixel indicates that the pixel is not foreground, that is, the pixel is background.

[0120] Compared with the entity feature of the pixel determined in step 620, the semantic feature of the pixel determined in step 630 is equivalent to compressing the feature of the pixel to a greater extent and classifying the pixel to a greater granularity, that is, classifying the pixel into two categories (foreground and non-foreground).

[0121] In some embodiments of the present application, step 630 comprises: performing feature upsampling on the shared feature of the second region image in stages to obtain second upsampled features of the second region image; and performing binary segmentation on the second upsampled features of the second region image to obtain semantic features of each pixel in the second region image.

[0122] In the present embodiment, the pixels in the second region image as foreground and the pixels as non-foreground (background) are distinguished through binary segmentation. In some embodiments of the present application, the binary segmentation can be performed by a connected component labeling method. The mask corresponding to the foreground is set to 1, and the mask of the non-foreground is set to 0. On this basis, each pixel is labeled as a connected component, and points connected to each other form a connected region.

[0123] Step 640: clustering each pixel in the second region image according to the entity feature of each pixel in the second region image and the semantic feature of each pixel in the second region image to determine a second entity pixel region in which at least one entity in the second region image is located.

[0124] In steps 620 and 630, both are pixel-level learning tasks, and in step 640, the pixels in the second region are segmented at the entity level. In specific embodiments, the entity-level segmentation can be achieved by clustering through a hierarchical clustering algorithm, a density clustering algorithm, etc. The density clustering algorithm is based on the density of the data set in the spatial distribution for clustering, and does not need to pre-set the number of clusters, so it is particularly suitable for clustering data sets with unknown content.

[0125] The density clustering algorithm, such as DBSCAN (Density-Based Spatial Clustering of Application with Noise), OPTICS (Ordering Points to identify the clustering structure) algorithm, etc., can be selected as needed.

[0126] It is worth mentioning that the above-mentioned embodiments for implementing entity segmentation are also applicable to entity segmentation of the first region image, and the process of entity segmentation of the first region image will not be described herein.

[0127] In some embodiments of the present application, the prediction of the spatial position information of the entity according to the shared features of the second region image can also be performed. Specifically, after step 610, the method further comprises: predicting the spatial position information of the entity in the second region image according to the shared features of the second region image. Wherein, the spatial position information of the entity can include at least one of the entity offset information of the entity and the height position information of the entity. The entity offset information includes the position and size of the base of the entity.

[0128] In some embodiments of the present application, the prediction of the entity offset information of the entity in the second region image can be performed by the process shown in Figure 7 As shown in Figure 7 As shown in

[0129] Step 710, performing feature up-sampling level by level according to the shared features of the second region image to obtain third up-sampling features of the second region image.

[0130] Step 720, performing offset position prediction according to the third up-sampling features of the second region image to obtain the position offset information corresponding to each pixel in the second region image.

[0131] Step 730, determining the entity offset information of the entity indicated by the second entity pixel region according to the position offset information corresponding to each pixel in the second entity pixel region in the second region image.

[0132] In the image, what is presented is the surface region of the entity under the corresponding image collection view angle, such as the top region of the entity, the side region of the entity, etc. Since it cannot be guaranteed that the image collection is performed under the collection view angle perpendicular to the base of the entity, therefore, the pixels in the surface region of the entity in the image are offset relative to the base of the entity. The position offset information corresponding to the pixel in step 720 is used to indicate the offset amount of the pixel relative to the base of the entity. In specific embodiments, the position offset information corresponding to each pixel can be predicted by the neural network after training (such as the offset amount prediction network below).

[0133] On the basis of obtaining the position offset information of each pixel in the second region image and the second entity pixel region where each entity in the second region image is located, the position offset information of each pixel in the same second entity pixel region can be correspondingly determined, therefore, the entity offset information of the entity indicated by the second entity pixel region is comprehensively determined according to the position offset information of each pixel in the same second entity pixel region.

[0134] In some embodiments of the present application, step 730 comprises: determining the offset median of the second entity pixel region according to the position offset information corresponding to each pixel in the second entity pixel region; and determining the offset median of the second entity pixel region as the entity offset information of the entity indicated by the second entity pixel region.

[0135] If the position offset information corresponding to a pixel indicates the position offset amount of the pixel in the x direction and the y direction, the corresponding offset median includes the offset median in the x direction and the offset median in the y direction. In other embodiments, the position offset amount of a set percentile (for example, 60%, 80%, etc.) can also be determined as the entity height information of the entity indicated by the second entity pixel region, which is not specifically limited here.

[0136] In some embodiments of the present application, the entity height of an entity can be determined by a process as shown in Figure 8 , including: Figure 8

[0137] Step 810: Gradually performing feature up-sampling according to the shared features of the second region image to obtain fourth up-sampled features of the second region image.

[0138] Step 820: Predicting the height information corresponding to each pixel in the second region image according to the fourth up-sampled features of the second region image.

[0139] The height information corresponding to a pixel is used to indicate the height of the pixel in the physical space relative to the base of the entity where the pixel is located. In specific embodiments, the height information corresponding to each pixel can be predicted by a neural network (for example, the height prediction network below) after training.

[0140] Step 830: Determining the entity height of the entity indicated by the second entity pixel region according to the height information corresponding to each pixel in the second entity pixel region in the second region image.

[0141] On the basis of obtaining the height information of each pixel in the second region image and the second entity pixel region where each entity in the second region image is located, the height information of each pixel in the same second entity pixel region can be correspondingly determined, and therefore, the entity height of the entity indicated by the same second entity pixel region is comprehensively determined according to the height information of each pixel in the same second entity pixel region.

[0142] In some embodiments of the present application, step 830 includes: determining the height median of the second entity pixel region according to the height information corresponding to each pixel in the second entity pixel region; and determining the height median of the second entity pixel region as the entity height of the entity indicated by the second entity pixel region.

[0143] In other embodiments, the height information of a set percentile (for example, 60%, 65%, 80%, 90%, etc.) can also be determined as the entity height of the entity indicated by the second entity pixel region, which is not specifically limited here.

[0144] ​In the above embodiment, the shared features of the second region image obtained in the sharing step 610 are shared, and the semantic features, entity features, position offset information and height information of each pixel in the second region image are determined, without respectively performing feature down-sampling on the second region image in the four processes respectively, so that the consumption of computing resources is saved in this way.

[0145] Similarly, the spatial position information (entity offset information and height position information) corresponding to each entity in the first region image can also be determined in the same way as the above embodiment, which will not be described here.

[0146] Figure 9 FIG. 9 is an architecture diagram of an entity segmentation network according to an embodiment of the present application, which can be used for entity segmentation of the first region image or the second region image. The entity segmentation network includes a shared feature extraction network 910, a height prediction network, an entity feature extraction network, a semantic feature extraction network, an offset prediction network and an entity clustering module 960.

[0147] The shared feature extraction network 910 is used for feature down-sampling of an input image to obtain shared features of the input image as basic data for subsequent semantic feature extraction, entity feature extraction, height prediction and position offset prediction. The input image can be from the first region image or the second region image.

[0148] The shared feature extraction network 910 includes a plurality of cascaded down-sampling modules, each of which includes a convolution layer, a normalization layer and an activation layer. The convolution layer of the bottom down-sampling module is responsible for extracting basic features such as edge texture of the input image, and the convolution layer of the high-level down-sampling module abstractly combines the basic texture features. The normalization layer performs normal distribution processing on the features output by the convolution layer. The activation layer is used for non-linear mapping of the features output by the normalization layer to enhance the generalization ability of the network.

[0149] The entity feature extraction network includes a first up-sampling network 931 and a pixel embedding network. The first up-sampling network 931 is used for feature up-sampling of the shared features output by the shared feature extraction network 910 to obtain first up-sampling features corresponding to the input image. The pixel embedding network is used for pixel embedding of the first up-sampling features output by the first up-sampling network 931 to obtain entity features of each pixel in the input image. Figure 9 FIG. 9B shows an entity feature map 932 representing the entity features of each pixel in the input image.

[0150] The first up-sampling network 931 includes a plurality of cascaded first up-sampling modules, and according to a feature pyramid fusion strategy, the input of each first up-sampling module includes not only the output of the previous first up-sampling module, but also a feature map of the same size output by a down-sampling module in the shared feature extraction network 910, and each first up-sampling module fuses the two input feature maps to obtain the input of the first up-sampling module at this stage, and the input is used for feature fusion by the next first up-sampling module.

[0151] In specific embodiments, the loss function of the entity feature extraction network can include an intra-class cohesion loss function and an inter-class distinction loss function, and the two loss functions are used to constrain the training of the entity feature extraction network 930.

[0152] The semantic feature extraction network includes a second up-sampling network 941 and a binary line segmentation network. The second up-sampling network 941 performs feature up-sampling on the shared features output by the shared feature extraction network 910 to obtain second up-sampling features corresponding to the input image; and the binary line segmentation network is used to perform binary line segmentation on the second up-sampling features output by the second up-sampling network 941 to obtain semantic features of each pixel in the input image. Figure 9 Fig. 6 shows a semantic feature map 942 representing semantic features of each pixel in an input image.

[0153] The second up-sampling network 941 in the semantic feature extraction network can include a plurality of cascaded second up-sampling modules, and the second up-sampling network 941 performs feature fusion in each second up-sampling module according to a feature pyramid strategy, i.e., in each second up-sampling module, a feature map output by the previous second up-sampling module and a feature map of the same size output by a down-sampling module in the shared feature extraction network 910 are fused, and the fusion result is taken as the output of the second up-sampling module at this stage. Then, binary line segmentation is performed on the feature map output by the last second up-sampling module. In specific embodiments, the feature map output by the last second up-sampling module in the second up-sampling network 941 has the same size as the input image. In specific embodiments, the semantic feature extraction network is trained for binary classification (foreground and non-foreground (background)) during the training process, and the loss function of the semantic feature extraction network can be a standard cross-entropy loss function.

[0154] The entity clustering network 960 is used to perform entity clustering according to the entity features of each pixel in the input image and the semantic features of each pixel to determine the entity pixel region where each entity in the input image is located. Figure 9 Fig. 7 shows an entity marked image 961 after the input image is subjected to entity segmentation, in which the entity pixel region where each entity in the input image is located is marked.

[0155] The offset prediction network comprises a third up-sampling network 951 configured to perform feature up-sampling on the shared features output by the shared feature extraction network 910 to obtain corresponding third up-sampled features, and then determine the entity offset information of each entity in the input image according to the third up-sampled features.

[0156] The third up-sampling network 951 comprises a plurality of cascaded third up-sampling modules. The third up-sampling network 951 performs feature fusion according to a feature pyramid strategy, i.e., each third up-sampling module fuses the feature map of the previous third up-sampling module and the feature map of the same size output by the down-sampling module in the shared feature extraction network 910, and takes the fusion result as the output of the third up-sampling module.

[0157] In this embodiment, the offset prediction network is a regression network, which predicts the position offset in the x direction and the y direction for each pixel in the input image. In a specific embodiment, the loss function of the offset prediction network can be an L2 least square loss function.

[0158] In the training image set corresponding to the offset prediction network, the position offset in the x direction and the y direction indicated by the offset label corresponding to the pixel that is not foreground or has no position offset in the training image is 0. The offset label of a pixel is used to indicate the real position offset of the corresponding pixel. In the training of the offset prediction network, the function value of the loss function is calculated according to the predicted position offset of the pixel output by the offset prediction network and the position offset indicated by the offset label of the pixel, and the offset prediction network is trained according to the function value of the loss function.

[0159] The height prediction network comprises a fourth up-sampling network 921 configured to perform feature up-sampling on the shared features output by the shared feature extraction network 910 to obtain corresponding fourth up-sampled features, and then determine the entity height of each entity in the input image according to the fourth up-sampled features.

[0160] The fourth up-sampling network 921 comprises a plurality of cascaded fourth up-sampling modules. The fourth up-sampling network 951 performs feature fusion according to a feature pyramid strategy, i.e., each fourth up-sampling module fuses the feature map of the previous fourth up-sampling module and the feature map of the same size output by the down-sampling module in the shared feature extraction network 910, and takes the fusion result as the output of the fourth up-sampling module. In this embodiment, the height prediction network is a regression network, and the loss function of the height prediction network can be an L1 absolute value loss function.

[0161] In the embodiment, the shared feature extraction network 910 forms a U-shaped network structure (U-net) with the first up-sampling network 931, the second up-sampling network 941, the third up-sampling network 951, and the fourth up-sampling network 921 respectively.

[0162] In some embodiments of the present application, after the step of predicting the spatial position information of the entity in the second region image according to the shared features of the second region image, the method further comprises: updating the model data of the entity in the map according to the spatial position information of the entity in the second region image and the change attribute of the entity in the second region image.

[0163] Satellite images are used in the production and updating of electronic maps. Satellite images can be taken by satellites on geographical regions on the ground and then transmitted to servers on the ground through antennas.

[0164] After an entity is identified to have changed according to a collected satellite image, the model data of the changed entity needs to be updated in the map data. In an electronic map, the model data of an entity is at least obtained by constructing an entity model according to the base information (size information and position information) of the entity; in a three-dimensional electronic map, the entity model construction further needs the entity height information of the entity. The base information (size information and position information) of the entity and the entity height information of the entity can be collectively referred to as the spatial position information of the entity.

[0165] If the collection time corresponding to the second image is later than the collection time of the first image, the entity on the corresponding position in the electronic map needs to be consistent with the entity on the corresponding position in the second image. Therefore, the model data of the corresponding entity in the map is updated according to the spatial position information of the entity in the second region image and the change attribute of the entity in the second region image.

[0166] Specifically, if the change attribute of the entity is new construction, the entity model of the newly constructed entity in the electronic map needs to be established according to the spatial position information of the newly constructed entity, and the established entity model is added to the corresponding position in the electronic map according to the position of the newly constructed entity in the geographical region. If the change attribute of the entity is demolition, the entity model of the entity needs to be deleted in the electronic map. If the change attribute of the entity is reconstruction, the entity model of the reconstructed entity in the electronic map needs to be reconstructed according to the spatial position information of the reconstructed entity, and the entity model at the position of the reconstructed entity in the electronic map is replaced by the reconstructed entity model. If the change attribute of the entity is unchanged, the model data corresponding to the entity in the electronic map does not need to be updated.

[0167] In the background data of the base map, it is the key to quickly find the changed part and update and maintain to ensure the freshness of the data. The existence and accuracy of the background data directly affect the application effect and user experience of the digital topographic map product. The method provided in the application can first determine the entity change area according to two satellite images collected at different times to ensure a high recall rate; and further perform entity segmentation and entity matching on the entity change area in the two satellite images, so as to further determine the change attribute of each entity in the entity change area, realize the detection of entity change in a smaller granularity, and ensure the accuracy of the determined change attribute of the entity. In the process, it can also identify whether the entity change area detected and determined in the entity change area detection stage is a false detection result. Then, the spatial position information and the corresponding change attribute of the entity to be changed are used to update the background data in the map in time, effectively ensuring the accuracy of the background data in the map.

[0168] In practice, the method of the application is applied to the electronic map product, the changed building entity can be quickly and accurately found, the accuracy rate reaches 90%+, the recall rate reaches 90%+, and the update efficiency of the background data in the map is greatly improved to ensure the data freshness of the background data in the map.

[0169] Figure 10 is a flowchart of the change attribute of the entity in the image according to an embodiment of the application. As shown in Figure 10 , specifically includes:

[0170] Step 1010: generating a task. Specifically, the client provides a page for selecting cities and regions, determines a target geographic area according to the selected city and the selected region, and then obtains satellite images collected for the target geographic area at different times to obtain an image pair including a first image and a second image. The change attribute detection task is generated according to the obtained first image and second image.

[0171] Step 1020, entity change area detection. In this step, after differentiating the first image and the second image, the entity change area detection is performed. If it is detected and determined that there is no entity change area, the process is ended; if it is detected and determined that there is an entity change area, step 1030 is executed.

[0172] Step 1030, extracting the entity change area. When it is detected and determined that there is an entity change area, the change area information of the detected entity change area is obtained, so that the region extraction is performed from the first image according to the change area information to obtain a first region image; and the region extraction is performed from the second image according to the change area information to obtain a second region image.

[0173] Step 1040, entity segmentation. In this step, the first region image and the second region image are respectively subjected to entity segmentation to determine the first entity pixel region in which each entity in the first region image is located and the second entity pixel region in which each entity in the second region image is located.

[0174] Step 1050, determining a changed entity and a changed attribute. In this step, the first entity pixel region and the second entity pixel region are subjected to entity matching, so as to determine whether the entity indicated by the first entity pixel region and the entity indicated by the second entity pixel region are the same entity. If not, it indicates that the entity at the corresponding position has changed. Conversely, if they are the same entity, it indicates that the entity at the corresponding position has not changed. If it is determined that the entity has changed, the entity change attribute is further determined according to the intersection-over-union of the first entity pixel region and the second entity pixel region.

[0175] Step 1060, correcting the map data. In this step, the spatial position information of the changed entity and the corresponding change attribute are submitted to a map data editing platform, and a technician updates the map data according to the spatial position information of the changed entity and the corresponding change attribute.

[0176] Figures 11-1 to 11-4 FIG. 6 is a schematic diagram showing the determination result of the change attribute of the building according to an embodiment of the present application. Figure 11-1 FIG. 7 shows a schematic diagram of the determination result of the change attribute of region A. There is no building bounding box in the detected output image pair (image 11-1-I and image 11-1-II), and there is actually no building in the determined region A. It can be determined that region A is a false detection result in the building change region detection stage. Correspondingly, in the subsequent process, the change attribute determination result of region A does not need to be submitted to the map data editing platform.

[0177] Figure 11-2 FIG. 8 shows a schematic diagram of the determination result of the change attribute of region B. Figure 11-2 FIG. 9 shows the building in region B in image 11-2-I and the building in region B in image 11-2-II. According to the method provided in the present application, it is determined that the building in region B in image 11-2-I is the same as the building in region B in image 11-1-I, i.e., the building in region B has not changed, i.e., the corresponding change attribute is unchanged.

[0178] Figure 11-3A schematic diagram showing the change attribute determination results for region C and region D is shown. The detection result shows that the building in region C in image 11-3-I is different from the building in region C shown in image 11-3-II, and therefore, the building in region C in image 11-3-I is a building being reconstructed. There are several buildings in region D in image 11-3-I collected at an earlier time, while no building is detected in region D in image 11-3-II collected at a later time, and therefore, the building in region D in image 11-3-I is a building being demolished, i.e., the corresponding change attribute is being demolished.

[0179] Figure 11-4 A schematic diagram showing the change attribute determination results for region E, region F and region G is shown. The detection result shows that there is no building in region E, region F and region G in image 11-4-I collected at an earlier time, while there are buildings in region E, region F and region G in image 11-4-II collected at a later time, and therefore, it is determined that the buildings in region E, region F and region G in image 11-4-II are all newly built buildings, i.e., the corresponding change attribute is being newly built.

[0180] As can be seen from the above, according to the method provided in the present application, in the entity segmentation and entity matching stage, the entity change detection stage can be identified as a misdetected entity change region, and therefore, even if the entity change region is misdetected due to reasons such as light, weather, seasonal change, etc., it can be accurately identified in the entity segmentation and entity matching stage, and the accuracy of the detected and determined change attribute of the entity is ensured.

[0181] The device embodiment of the present application is introduced below, which can be used to execute the method in the above-mentioned embodiments of the present application. For details not disclosed in the device embodiment of the present application, please refer to the above-mentioned method embodiment of the present application.

[0182] Figure 12is a block diagram of an entity detection apparatus according to an embodiment of the present application, which comprises: an image acquisition module 1210 configured to acquire a first image and a second image, the first image corresponding to a first acquisition time and the second image corresponding to a second acquisition time different from the first acquisition time; a change region detection module 1220 configured to detect an entity change region of the second image relative to the first image to obtain change region information; a region extraction module 1230 configured to extract a region from the first image according to the change region information to obtain a first region image, and extract a region from the second image according to the change region information to obtain a second region image; a first entity segmentation module 1240 configured to perform entity segmentation on the first region image to obtain a first entity pixel region in which at least one entity in the first region image is located; a second entity segmentation module 1250 configured to perform entity segmentation on the second region image to obtain a second entity pixel region in which at least one entity in the second region image is located; and a change attribute determination module 1260 configured to perform entity matching according to the first entity pixel region and the second entity pixel region to determine a change attribute of the entity between the first image and the second image.

[0183] In some embodiments of the present application, the change region detection module 1220 comprises: a difference unit configured to perform difference between the first image and the second image to obtain a difference image; a first feature down-sampling unit configured to perform feature down-sampling on the difference image step by step to obtain at least two levels of first down-sampled feature maps; a feature pyramid fusion unit configured to perform feature pyramid fusion according to the at least two levels of first down-sampled feature maps to obtain feature maps at at least two scales; and a change region prediction unit configured to perform entity change region detection on the feature maps at the at least two scales respectively to obtain at least two groups of reference change region information; and a non-maximum suppression unit configured to perform non-maximum suppression according to the change regions respectively indicated by the at least two groups of reference change region information to obtain the change region information.

[0184] In some embodiments of the present application, the feature maps at the at least two scales comprise a first feature map, at least one level of second feature maps and at least one level of third feature maps; the feature pyramid fusion unit comprises: a first feature map determination unit configured to take a down-sampled feature map with a maximum down-sampling level among the at least two levels of down-sampled feature maps as the first feature map; a convolution processing unit configured to perform convolution processing on the first feature map step by step to obtain at least one level of second feature maps; a first feature up-sampling unit configured to perform up-sampling on the first feature map to obtain a first up-sampled feature map; and a fusion unit configured to fuse the first up-sampled feature map and a first down-sampled feature map with a same size as the first up-sampled feature map to obtain an initial third feature map, and update a fusion frequency; if the fusion frequency does not reach a fusion frequency threshold, continue to perform down-sampling and fusion according to the initial third feature map, and update the fusion frequency, until the fusion frequency reaches the fusion frequency threshold.

[0185] In some embodiments of the present application, the second entity segmentation module 1250 comprises: a second feature down-sampling unit configured to perform feature down-sampling on the second region image level by level to obtain shared features of the second region image; an entity feature extraction unit configured to perform entity feature extraction according to the shared features of the second region image to obtain entity features of each pixel in the second region image; a semantic feature extraction unit configured to perform semantic feature extraction according to the shared features of the second region image to obtain semantic features of each pixel in the second region image, wherein the semantic features are used to indicate whether the corresponding pixel is foreground; and a clustering unit configured to perform clustering according to the entity features of each pixel in the second region image and the semantic features of each pixel in the second region image to determine a second entity pixel region in which at least one entity in the second region image is located.

[0186] In some embodiments of the present application, the entity feature extraction unit comprises: a second feature up-sampling unit configured to perform feature up-sampling on the shared features of the second region image level by level to obtain first up-sampling features of the second region image; and a pixel embedding unit configured to perform pixel embedding according to the first up-sampling features of the second region image to obtain the entity features of each pixel in the second region image.

[0187] In some embodiments of the present application, the semantic feature extraction unit comprises: a third feature up-sampling unit configured to perform feature up-sampling on the shared features of the second region image level by level to obtain second up-sampling features of the second region image; and a binary segmentation unit configured to perform binary segmentation according to the second up-sampling features of the second region image to obtain the semantic features of each pixel in the second region image.

[0188] In some embodiments of the present application, the entity detection apparatus further comprises a spatial position information determination module configured to predict spatial position information of an entity in the second region image according to the shared features of the second region image.

[0189] In some embodiments of the present application, the spatial position information comprises entity offset information; and the spatial position information determination module comprises: a fourth feature up-sampling unit configured to perform feature up-sampling on the shared features of the second region image level by level to obtain third up-sampling features of the second region image; an offset position prediction unit configured to perform offset position prediction according to the third up-sampling features of the second region image to obtain position offset information corresponding to each pixel in the second region image; and an entity offset information determination unit configured to determine entity offset information of an entity indicated by the second entity pixel region according to the position offset information corresponding to each pixel in the second entity pixel region in the second region image.

[0190] In some embodiments of the present application, the spatial position information comprises entity height, and the spatial position information determining module comprises: a fifth feature up-sampling unit configured to perform feature up-sampling on the shared features of the second region image in stages to obtain fourth up-sampled features of the second region image; a height information predicting unit configured to predict height information corresponding to each pixel in the second region image according to the fourth up-sampled features of the second region image; and an entity height determining unit configured to determine the entity height of the entity indicated by the second entity pixel region according to the height information corresponding to each pixel in the second entity pixel region in the second region image.

[0191] In some embodiments of the present application, the entity height determining unit comprises: a height median determining unit configured to determine the height median of the second entity pixel region according to the height information corresponding to each pixel in the second entity pixel region; and a determining unit configured to determine the height median of the second entity pixel region as the entity height of the entity indicated by the second entity pixel region.

[0192] In some embodiments of the present application, the entity change attribute determining apparatus further comprises: an updating module configured to update the model data of the entity in the map according to the spatial position information of the entity in the second region image and the change attribute of the entity in the second region image.

[0193] In some embodiments of the present application, the change attribute determining module 1260 comprises: an intersection over union calculating unit configured to determine the intersection over union of the first entity pixel region and the second entity pixel region according to the first entity pixel region and the second entity pixel region; and a change attribute determining unit configured to determine the change attribute corresponding to the intersection over union as the change attribute of the entity between the first image and the second image.

[0194] Figure 13 A structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application is shown. It should be noted that, Figure 13 The computer system 1300 of the electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0195] As Figure 13As shown, the computer system 1300 includes a central processing unit (CPU) 1301 which can execute various appropriate actions and processes in accordance with programs stored in a read-only memory (ROM) 1302 or loaded from the storage section 1308 into a random access memory (RAM) 1303, such as executing the methods in the above-described embodiments. Various programs and data required for system operation are also stored in the RAM 1303. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other through a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0196] Connected to the I / O interface 1305 are an input section 1306 including a keyboard, a mouse, etc.; an output section 1307 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as necessary. A removable recording medium 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1310 as necessary, so that a computer program read therefrom is installed into the storage section 1308 as necessary.

[0197] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 1309, and / or installed from the removable recording medium 1311. When the computer program is executed by the central processing unit (CPU) 1301, various functions defined in the systems of the present application are executed.

[0198] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer-readable signal medium can include a data signal carrying computer-readable program code in a baseband or as a part of a carrier wave. Such a propagated data signal can take on various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium that can send, propagate or transmit a program for use by or in connection with an instruction execution system, device or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0199] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In the flowcharts or block diagrams, each block can represent a module, a program segment or a part of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the involved functions. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0200] The units described in the embodiments of the present application can be implemented by software, or can be implemented by hardware, and the units described can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0201] As another aspect, the present application also provides a computer readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The computer readable storage medium carries computer readable instructions, which, when executed by a processor, implement the method in any of the above embodiments.

[0202] According to an aspect of the present application, an electronic device is also provided, which includes a processor, and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method in any of the above embodiments.

[0203] According to an aspect of the embodiments of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method in any of the above embodiments.

[0204] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into a plurality of modules or units.

[0205] From the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or network, and includes a number of instructions to make a computing device (which can be a personal computer, server, touch terminal, or network device, etc.) execute the methods according to the embodiments of the present application.

[0206] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the application embrace any and all variations of the present application that fall within the scope of the general inventive concept as defined by the appended claims and their equivalents. It is intended that the application encompass all such variations as fall within the scope of the appended claims and their equivalents.

[0207] It is to be understood that the application is not limited to the precise construction herein disclosed and shown in the drawings, and that various changes in shape, size and materials can be made to the application without departing from the scope of the application. The scope of the application is only limited by the appended claims.

Claims

1. An entity detection method, characterized by, The method comprises: acquiring a first image and a second image, the acquisition time corresponding to the first image being different from the acquisition time corresponding to the second image; detecting an entity change region of the second image relative to the first image to obtain change region information; performing region extraction on the first image according to the change region information to obtain a first region image, and performing region extraction on the second image to obtain a second region image; performing entity segmentation on the first region image to obtain a first entity pixel region in which at least one entity in the first region image is located, and performing entity segmentation on the second region image to obtain a second entity pixel region in which at least one entity in the second region image is located; performing entity matching according to the first entity pixel region and the second entity pixel region to determine a change attribute of the at least one entity between the first image and the second image.

2. The method of claim 1, wherein, The detection of the entity change region of the second image relative to the first image to obtain the change region information comprises: performing difference on the first image and the second image to obtain a difference image; performing feature down-sampling on the difference image step by step to obtain at least two levels of first down-sampling feature maps; performing feature pyramid fusion on the at least two levels of first down-sampling feature maps to obtain feature maps at at least two scales; performing entity change region detection on the feature maps at the at least two scales respectively to obtain at least two groups of reference change region information; performing non-maximum suppression according to the change regions respectively indicated by the at least two groups of reference change region information to obtain the change region information.

3. The method of claim 2, wherein, The feature maps at the at least two scales comprise a first feature map, at least one level of second feature maps, and at least one level of third feature maps. The feature pyramid fusion on the at least two levels of first down-sampling feature maps to obtain the feature maps at the at least two scales comprises: taking a down-sampling feature map corresponding to the largest down-sampling level in the at least two levels of down-sampling feature maps as the first feature map; performing convolution processing on the first feature map step by step to obtain the at least one level of second feature maps; performing up-sampling on the first feature map to obtain a first up-sampling feature map; performing fusion on the first up-sampling feature map and a first down-sampling feature map with the same size as the first up-sampling feature map to obtain an initial third feature map, and updating a fusion frequency; if the fusion frequency does not reach a fusion frequency threshold, continuing to perform down-sampling and fusion according to the initial third feature map, and updating the fusion frequency until the fusion frequency reaches the fusion frequency threshold.

4. The method of claim 1, wherein, The entity segmentation on the second region image to obtain the second entity pixel region in which at least one entity in the second region image is located comprises: performing feature down-sampling on the second region image step by step to obtain shared features of the second region image; performing entity feature extraction according to the shared features of the second region image to obtain entity features of each pixel in the second region image; extracting semantic features according to the shared features of the second region image, to obtain semantic features of each pixel in the second region image; the semantic features are used to indicate whether the corresponding pixel is foreground or not; performing clustering according to the entity features of each pixel in the second region image and the semantic features of each pixel in the second region image, to determine a second entity pixel region in which at least one entity in the second region image is located.

5. The method of claim 4, wherein, The extracting entity features according to the shared features of the second region image, to obtain entity features of each pixel in the second region image, comprises: performing feature upsampling on the shared features of the second region image step by step, to obtain first upsampling features of the second region image; performing pixel embedding according to the first upsampling features of the second region image, to obtain entity features of each pixel in the second region image.

6. The method of claim 4, wherein, The extracting semantic features according to the shared features of the second region image, to obtain semantic features of each pixel in the second region image, comprises: performing feature upsampling on the shared features of the second region image step by step, to obtain second upsampling features of the second region image; performing binary segmentation according to the second upsampling features of the second region image, to obtain semantic features of each pixel in the second region image.

7. The method of claim 4, wherein, After the step of performing feature downsampling on the second region image step by step to obtain shared features of the second region image, the method further comprises: predicting spatial position information of entities in the second region image according to the shared features of the second region image.

8. The method of claim 7, wherein, The spatial position information comprises entity offset information; the predicting spatial position information of entities in the second region image according to the shared features of the second region image comprises: performing feature upsampling on the shared features of the second region image step by step, to obtain third upsampling features of the second region image; performing offset position prediction according to the third upsampling features of the second region image, to obtain position offset information corresponding to each pixel in the second region image; determining entity offset information of an entity indicated by the second entity pixel region according to the position offset information corresponding to each pixel in the second entity pixel region in the second region image.

9. The method according to claim 7 or 8, characterized in that, The spatial position information comprises entity height; the predicting spatial position information of entities in the second region image according to the shared features of the second region image comprises: performing feature upsampling on the shared features of the second region image step by step, to obtain fourth upsampling features of the second region image; predicting height information corresponding to each pixel in the second region image according to the fourth upsampling features of the second region image; determining entity height of an entity indicated by the second entity pixel region according to the height information corresponding to each pixel in the second entity pixel region in the second region image.

10. The method of claim 9, wherein, The determining entity height of an entity indicated by the second entity pixel region according to the height information corresponding to each pixel in the second entity pixel region in the second region image comprises: determine a median height of the second entity pixel region according to height information corresponding to each pixel in the second entity pixel region; determine the median height of the second entity pixel region as the entity height of the entity indicated by the second entity pixel region.

11. The method of claim 7, wherein, After the spatial position information of the entity in the second region image is predicted according to the shared features of the second region image, the method further comprises: update model data of the entity in the map according to the spatial position information of the entity in the second region image and the change attribute of the entity in the second region image.

12. The method of claim 1, wherein, The entity matching is performed according to the first entity pixel region and the second entity pixel region, and the change attribute of the at least one entity between the first image and the second image is determined, comprising: determine an intersection-over-union of the first entity pixel region and the second entity pixel region according to the first entity pixel region and the second entity pixel region; determine the change attribute corresponding to the intersection-over-union as the change attribute of the corresponding entity between the first image and the second image.

13. An entity detection apparatus, characterized by comprise: an image acquisition module, configured to acquire a first image and a second image, the acquisition time corresponding to the first image being different from the acquisition time corresponding to the second image; a change region detection module, configured to detect an entity change region of the second image relative to the first image to obtain change region information; a region extraction module, configured to extract a first region image from the first image and a second region image from the second image according to the change region information; a first entity segmentation module, configured to perform entity segmentation on the first region image to obtain a first entity pixel region in which at least one entity in the first region image is located; a second entity segmentation module, configured to perform entity segmentation on the second region image to obtain a second entity pixel region in which at least one entity in the second region image is located; a change attribute determination module, configured to perform entity matching according to the first entity pixel region and the second entity pixel region to determine a change attribute of the at least one entity between the first image and the second image.

14. An electronic device, comprising: comprise: a processor; a memory, the memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method according to any one of claims 1-12.

15. A computer readable storage medium having computer readable instructions stored thereon, the computer readable instructions being executed by a processor to implement the method according to any one of claims 1-12.

Images