Method of identifying object region of image, electronic device and computer program product

By detecting image edges and semantic segmentation to identify object areas in video, the problem of difficult to automatically identify and edit image areas in the prior art is solved, and more efficient and flexible video editing is achieved.

CN119068381APending Publication Date: 2024-12-03DELL PROD LP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310652243.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-02
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is difficult to automatically identify and edit image areas, especially graphic elements, in videos, limiting the applicability and efficiency of video editing.

Method used

By detecting edges in images and segmenting the image into multiple semantic regions based on semantic segmentation, combining edges and semantic regions to determine the object region, and using multiple models (text recognition, edge detection, semantic segmentation, residual network, etc.) to identify the image in the image object area.

Benefits of technology

It realizes precise identification and editing of graphic element objects in the image, improves the efficiency and flexibility of video processing, and reduces the demand for computing resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068381A_ABST
    Figure CN119068381A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a method for identifying an object area in an image, electronic equipment and a computer program product. The method includes detecting edges in an image based on pixels in the image. The method further includes segmenting the image into a plurality of semantic regions based on semantics of the object in the image. The method further includes identifying an object region covering the object based on the edge and the plurality of semantic regions. In this way, the position of the object comprising the graphic element in the image and the label of the object can be automatically recognized, and the efficiency of subsequent processing of the image is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computers, and more particularly, to a method for identifying an object region of an image, an electronic device, and a computer program product. Background Art

[0002] Today, videos are the most common and popular data on the Internet, and are widely used in many fields such as entertainment, production, and communication. In many cases, users want to edit videos to create the desired video content. For example, some users may want to change the trademark in the video, block some advertisements, and update the TV logo. To do this, it is necessary to identify the trademark in all image frames in the video, then remove the trademark and replace it with a new one. Summary of the invention

[0003] An embodiment of the present disclosure provides a solution for identifying an object region in an image.

[0004] In a first aspect of the present disclosure, a method for identifying an object region in an image is provided. The method includes detecting edges in the image based on pixels in the image. The method also includes segmenting the image into a plurality of semantic regions based on the semantics of the object in the image. The method also includes identifying an object region covering the object based on the edge and the plurality of semantic regions.

[0005] In a second aspect of the present disclosure, an electronic device is provided. The electronic device includes a processor; and a volatile memory coupled to the processor, the volatile memory having instructions stored therein, which, when executed by the processor, cause the device to perform an action. The action includes detecting edges in an image based on pixels in the image. The action also includes segmenting the image into a plurality of semantic regions based on the semantics of an object in the image. The action also includes identifying an object region covering an object based on the edge and the plurality of semantic regions.

[0006] In a third aspect of the present disclosure, there is provided a computer program product tangibly stored on a computer readable medium and comprising machine executable instructions which, when executed, cause a machine to perform the method according to the first aspect.

[0007] This Summary is provided to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the exemplary embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present disclosure. In the accompanying drawings: Figure 1 A schematic diagram illustrating an example environment in which embodiments of the present disclosure may be implemented is shown.

[0009] Figure 2 A flowchart illustrating an example method of identifying an object region in an image according to an embodiment of the present disclosure;

[0010] Figure 3A A schematic diagram showing an example process of identifying an object region in an image according to an embodiment of the present disclosure;

[0011] Figure 3B A comparison diagram of the identification results obtained using different methods is shown;

[0012] Figure 4A-4B A schematic diagram showing an example process of creating content according to an embodiment of the present disclosure;

[0013] Figure 5 A schematic diagram illustrating an example process of annotating an image frame according to an embodiment of the present disclosure; and

[0014] Figure 6 A block diagram of an example device that may be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0015] The principles of the present disclosure will be described below with reference to several example embodiments shown in the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that these embodiments are described only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

[0016] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] As discussed above, when a user wants to edit an object in a video, he needs to select an area in the image that covers the target object. Editing videos and creating new content usually requires professional video editing technology. In conventional solutions, video creators can use text recognition models to identify text areas in image frames of videos, but cannot detect image areas in image frames. This greatly limits its applicability.

[0018] In view of this, an embodiment of the present disclosure proposes a solution for identifying object regions in an image to solve one or more of the above problems and other potential problems. In the solution, the edges in the image are first detected, and the image is segmented into multiple semantic regions based on the semantics of the objects in the image. Thereafter, the edges and semantic regions are integrated to determine the object region covering the object in the image. In this way, the position of the object including the graphic element in the image can be automatically identified, which improves the efficiency of subsequent processing of the image.

[0019] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure may be implemented. Figure 1 As shown, the example environment 100 includes a computing device 110 and a server 160. For example, an input video 120 including a plurality of image frames from a user can be input into the computing device 110. The computing device 110 can extract the image frames in the input video 120 and perform image processing on the image frames. On the other hand, the input video 120 can also be input into a video encoder 141 and converted into code data 151 by the video encoder 141.

[0020] exist Figure 1 In the illustrated embodiment, after the image frame of the input video 120 is processed by the computing device 110, a processed image frame 131 in which the object area is segmented and attribute data 132 of the object in the video can be obtained. For example, motorcycles, lanes, etc. under the object category, and brand 1, etc. under the sign category appear in the image frame. The attribute data 132 includes the coordinates of the motorcycle labeled "000" and its corresponding object area. The attribute data 132 also includes the coordinates of the lane labeled "001" and its corresponding object area. In addition, the attribute data 132 also includes the coordinates of the brand 1 labeled "0000" and its corresponding object area. Brand 1 is, for example, identified based on the text on the billboard in the image frame. The attribute data 132 is then input into the attribute encoder 142 to convert the attribute into code data 152 that is easy to store. Both the code data 151 and the code data 152 are stored in the server 160 for subsequent editing.

[0021] Figure 2FIG. 2 is a flowchart showing an example method 200 for identifying an object region in an image according to an embodiment of the present disclosure. The method 200 may be performed by, for example Figure 1 Executed by the computing device 110 in.

[0022] At 202, computing device 110 detects edges in an image based on pixels in the image. The image is, for example, an image frame in a video input by a user. In image processing, an edge can be a curve whose trend is consistent with a path of rapid intensity changes in the image, that is, the difference between pixels on both sides of the edge is large, for example, the difference between pixels on both sides of the edge is greater than a certain threshold. Edges are generally associated with boundaries of objects in a scene. In some embodiments, when identifying edges in an image, computing device 110 can detect edges in the image based on the values ​​of pixels in the image according to a trained edge detection model.

[0023] At 204, the computing device 110 segments the image into a plurality of semantic regions based on the semantics of the objects in the image. Semantic segmentation is classification at the pixel level. That is, pixels belonging to the same class are classified into one class, so semantic segmentation is to understand the image at the pixel level. For example, Figure 1 The pixels belonging to people in the image are divided into one category, the pixels belonging to motorcycles are also divided into one category, and the pixels belonging to the corresponding objects in the image are also divided into one category. In addition, the background pixels are also divided into one category. It should be noted that semantic segmentation is different from instance segmentation. For example, if there are multiple people in a photo, for semantic segmentation, it is only necessary to classify the pixels of all people into one category, but instance segmentation also classifies the pixels of different people into different categories. In other words, instance segmentation goes a step further than semantic segmentation. However, here, semantic segmentation is sufficient to implement the scheme according to the embodiment of the present disclosure, thereby reducing the computational cost to a certain extent, and thus being able to identify image frames in the video stream in real time.

[0024] In some embodiments, when segmenting an image into multiple semantic regions, the computing device 110 segments the image into multiple semantic regions based on the category of the object and according to the semantic segmentation model, so that each obtained semantic region may have a contour associated with the corresponding object.

[0025] At 206, the computing device 110 identifies the object region covering the object based on the edge and the plurality of semantic regions. By comprehensively considering the edge and the corresponding semantic region, a semantic region including a graphic edge associated with the object can be determined. In this way, objects with graphic elements can be identified through edge detection and semantic segmentation, which has higher flexibility and applicability than existing text recognition detection, and has lower complexity and computing resource requirements than existing fine segmentation mechanisms.

[0026] In some embodiments, when identifying the object region, the computing device 110 may extract edge features of the edge and semantic features of the plurality of semantic regions, and then concatenate the edge features and the semantic features into global features. Finally, the computing device 110 may identify the object region based on the global features according to the trained classification model. Figure 3A-3B The specific process of identifying object areas in images is described in detail.

[0027] Figure 3A FIG. 3 is a schematic diagram showing an example process 301 of identifying an object region in an image according to an embodiment of the present disclosure. Figure 3A As shown, process 301 may be performed by a corresponding module or model of computing device 110. Computing device 110 receives image 310 from a user. Image 310 includes object 311. Object 311 is a logo of the trademark "PUMBA". Afterwards, image 310 is input into three models in parallel for processing.

[0028] In one branch, the image 310 is input into the text recognition model 321 to detect the text in the image 310. In some embodiments, the text recognition model 321 can be any model capable of performing text recognition functions, such as an optical character recognition (OCR) model. After being processed by the text recognition model 321, an image 331 is obtained. Multiple text regions in the image 331 are surrounded by bounding boxes. Figure 3A In the illustrated embodiment, the graphic in the “PUMBA” logo is not detected, and only the text “PUMBA” 3311 in the logo is provided with a bounding box. The image 331 is input into the residual network model 351 .

[0029] In another branch, the image 310 is input into the edge detection model 322 to detect the edges in the image 310. In some embodiments, the edge detection model 322 can be any model capable of performing edge detection functions, such as an edge detection model. After being processed by the edge detection model 322, an image 332 is obtained. The image 332 only includes edges without other colors. The edges of the graphics and text in the logo 3321 of "PUMBA" are detected and presented.

[0030] After obtaining the image 332, the image 332 is input to the residual network model 341. After being processed by the residual network model 341, the features of the image 332 are extracted. Thereafter, the extracted features are input to the morphological transformation model 342 (morphological operator). The morphological transformation model 342 can, for example, perform corrosion and expansion operations on the image 332 to make the edges continuous and closed. Here, the morphological transformation can be expressed as:

[0031] Where b(x) = ReLU(|x|, θ) (1)

[0032] Where x represents the feature; represents the expansion processing of V, ReLU represents the activation function; θ represents the learnable threshold.

[0033] In this way, denoising of the image 332 is achieved. The denoised image 332 is input into the residual network model 351.

[0034] In the last branch, the image 310 is input into the semantic segmentation model 323 to perform semantic segmentation on the image 310. Here, the indication 370 for the trademark is also input into the semantic segmentation model 323. The indication 370 can improve the directionality of the semantic segmentation and promote the segmentation of the region of interest. In some embodiments, the semantic segmentation model 323 can be a language-driven semantic segmentation model (Language-driven Semantic Segmentation, LSeg), or any model that can perform the semantic segmentation function. After being processed by the semantic segmentation model 323, an image 333 is obtained. Objects corresponding to different semantics in the image 333 are separated, for example, represented by different filling patterns (not shown in detail here). It can be seen that the logo 3331 of "PUMBA" is determined to have the semantic "trademark" and is filled with the same pattern as other trademarks. Afterwards, the image 333 is input into the residual network model 351.

[0035] The image data processed by each branch or the feature map corresponding to the image is input into the residual network model 351. In the residual network model 351, the features of the image 331, the features of the denoised image 332, and the features of the image 333 are first concatenated into global features and used as input. In this way, an image 310 has different features in three dimensions after passing through three different models. The result processed by the residual network model 351 is input into the normalized indicator function 352. The normalized indicator function 352 can be, for example, a softmax function. Here, the residual network model 351 and the normalized indicator function 352 can be regarded as a classification model for performing recognition of objects in the image 310. After being processed by the normalized indicator function 352, an output result 360 is obtained. The output result 360 includes, for example, an object area 361 of the logo 311 of the object "PUMBA" in the image 310. Here, the object area 361 can be represented by a set of coordinates and used as a mask to perform subsequent operations. Furthermore, the output result 360 also includes the name of the trademark “PUMBA” in the object area 361 .

[0036] exist Figure 3A In the illustrated embodiment, the computing device 110 includes a text recognition model 321, an edge detection model 322, a semantic segmentation model 323, a residual network model 341, a morphological transformation model 342, a residual network model 351, and a normalized indicator function 352. The various models collaborate to form an overall architecture for object region detection. The overall architecture can be represented as:

[0037] y=softmax(R(concat[D(V),M(R(E(V))),L(V)])) (2)

[0038] Among them, V represents the input video; D represents text recognition processing; E represents edge detection processing; R represents residual network processing; M represents morphological transformation processing; L represents semantic segmentation processing; concat represents concatenation processing; and softmax represents normalization indication processing.

[0039] The text recognition model 321, the edge detection model 322, and the semantic segmentation model 323 can use fixed parameters without training. In contrast, the residual network model 341, the morphological transformation model 342, and the residual network model 351 are trained to implement the object region identification mechanism according to the embodiments of the present disclosure.

[0040] Figure 3B FIG. 3 is a schematic diagram showing a comparison 303 between an original image and a marking result obtained using different methods. Figure 3BAs shown, the original image 310 is identified as image 370 after only semantic segmentation. Image 370 includes multiple objects with different semantics distinguished by different colors. However, the trademark on the billboard on the side of the track is blurred and cannot be distinguished. The original image 310 is identified as image 380 after only text recognition. The text in image 380 is surrounded by a bounding box to form text areas 381, 382, ​​383, 384, 385, and 386. In this implementation, text areas 382, ​​383, and 384 are only part of the corresponding trademark. For example, object 311 includes an image part in addition to the text part, but the image part of object 311 is not included in the bounding box in image 380. Similarly, the logo corresponding to text area 382 has a solid background in addition to the text "Donkey", and the background is not included in the bounding box.

[0041] After the object region identification mechanism of the embodiment of the present disclosure is applied to the original image 310, the identification result is image 390. Image 390 includes object regions 391, 392, 393, 394, 395, 396, and 397 corresponding to the trademark. In image 390, object regions 393, 394, and 395 completely cover the object, accurately identifying the complete trademark in the image.

[0042] After the input image is processed to identify the object regions of each object of the input image, the obtained labels, coordinates and masks of each object region can be used for image editing and content manipulation. Figure 4A FIG. 4 is a schematic diagram showing an example process 401 of creating content according to an embodiment of the present disclosure. The example process 401 may be, for example, Figure 1 The computing device 110 in FIG. Figure 4A As described above, computing device 110 receives input video 410 from user 450. The image frame of input video 410 includes a logo 411 of the trademark "PUMBA". Input video 410 is input to object recognition module 420 of computing device 110. Object recognition module 420 may, for example, perform the following operations: Figure 2 The method for identifying an object area in an image as shown, or comprising Figure 3A The text recognition model 321, edge detection model 322, semantic segmentation model 323, residual network model 341, morphological transformation model 342, residual network model 351 and normalization indicator function 352 are executed Figure 3A The process of identifying object regions in an image is shown.

[0043] The object recognition module 420 extracts image frames from the input video 410 and processes each image frame to obtain an execution result 430. The execution result 430 includes, for example, the coordinates of the logo 411 and the label trademark "PUMBA", as well as the corresponding mask. The execution result 430 is input into the content creation module 440. Here, the user 450 inputs a replacement request 461 for the trademark "PUMBA" to the computing device 110. The replacement request 461 indicates that the "PUMBA" logo 411 is replaced with the "PUDEL" logo. After receiving the replacement request 461, the content creation module 440 removes the object area corresponding to the "PUMBA" logo 411 in each image frame of the input video 410 using the mask in the execution result, thereby obtaining an image 410'. The image 410' includes a blank area 412. Afterwards, the content creation module 440 fills the blank area 412 with a "PUDEL" logo 413 having a shape corresponding to the blank area 412 to obtain the input video 410". In this way, the user 450 can achieve the desired operation only by providing the input video 410 and inputting an operation request for the input video 410. In this way, the intelligence of video editing is improved and the user experience is greatly improved.

[0044] Figure 4B FIG. 4 is a schematic diagram showing an example process 403 of creating content according to some other embodiments of the present disclosure. The example process 403 may be, for example, Figure 1 The computing device 110 in FIG. Figure 4B As shown, the content creation module 440 can be constructed based on a latent stable diffusion (LSD) model. Here, the user 450 inputs an image 471 and indicates to replace the content of the target region 472 of interest in the image 471. The mask of the target region 472 can be used Figure 3A 473. Image 473 is input into LSD model 481. In addition, user 450 also inputs user request 462 for editing the image into LSD model 481. For example, user 450 requests to add a house to the target area. After LSD model 481 performs the corresponding operation, the data is output to gated convolution module 482 for further processing. Gated convolution model 482 uses images and masks to guide the convolution process, so it can ensure smooth and consistent structural changes of the entire image. After processing by gated convolution model 482, content image 490 is created. Content image 490 includes a house covering the corresponding area.

[0045] When there is a lack of training data for a specific field, such as the current lack of corresponding training sets for trademark detection, in order to train a new machine learning architecture, a large amount of manually annotated training data needs to be collected. In order to save manpower costs, a mechanism is proposed here to use two discontinuous image frames in the video to annotate other image frames between the two image frames. Figure 5 FIG. 5 is a schematic diagram showing an example process 500 for annotating an image frame according to an embodiment of the present disclosure. The process 500 is, for example, Figure 1 The computing device 110 in FIG. Figure 5 As shown, the computing device 110 receives an input video including an image frame 510 and an image frame 520. Here, the image frame 510 is earlier than the image frame 520 in time, and a plurality of other image frames are also included between the image frame 510 and the image frame 520. The image frame 510 includes a logo 511 of the trademark "abibas" located at a first position, and the image frame 520 includes a logo 521 of the trademark "abibas" located at a second position. Since the logo 521 has only a text area, the image frame 510 and the image frame 520 are input into the text recognition module 540 to identify the position of the logo in the image frame. It should be understood that the logo of the trademark "abibas" as an identification object is only exemplary, and in addition to text, the identification object may also include graphics, and when the identification object has a graphic element, the image frame may be input into the semantic segmentation module for identification. After being processed by the text recognition module 540, an image frame 510' in which the logo is identified by a bounding box 531 and an image frame 520' in which the logo is identified by a bounding box 532 are obtained. In the embodiment described, since the time interval between the image frame 510 and the image frame 520 is short, the displacement of the logo 511 relative to the logo 521 can be regarded as being caused by the linear motion of the logo. Afterwards, the image frame 510' and the image frame 520' are input into the path interpolation module 550. The path interpolation module 550 can determine the position of the logo of the trademark "abibas" in multiple image frames between the image frame 510 and the image frame 520 based on the position of the bounding box 531 in the image frame 510', the position of the bounding box 532 in the image frame 520' and the linear motion path by the following relationship:

[0046] t i =t s +(is)(t e -t s )×N (3)

[0047] Wherein, i represents the i-th image frame; s represents the s-th image frame as the initial image frame (corresponding to image frame 510); e represents the e-th image frame as the last image frame (corresponding to image frame 520); t sThe coordinates of the logo representing the trademark "abibas" in the sth image frame; t e The coordinates of the logo representing the trademark "abibas" in the e-th image frame; t i The coordinates of the logo representing the trademark "abibas" in the i-th image frame, and N represents the number of image frames. Through formula (4), the path interpolation module 550 can determine the position of the logo in the second, third, fourth and fifth image frames. The obtained position and the name of the logo are used as annotations for the six image frames and form part of the training set 560 together with the six image frames. Figure 5 In the embodiment shown, the user only needs to annotate the key frames, and the module can autonomously find the path and apply linear interpolation to determine the annotation content of the image frames between the key frames. In this way, the annotation time can be significantly reduced and a large amount of annotation data can be created.

[0048] Figure 6 Schematic block diagram of an example device 600 that can be used to implement embodiments of the present disclosure is shown. Figure 6 As shown, the device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 602 or loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0049] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0050] The various processes and processing described above, such as method 200, may be performed by processing unit 601. For example, in some embodiments, method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by CPU 601, one or more actions of methods 200 and 700 described above may be performed.

[0051] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0052] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination of the above. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0053] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0054] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0055] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0056] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0057] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0058] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0059] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for identifying an object region in an image, comprising: Detecting edges in the image based on pixels in the image; Based on the semantics of objects in the image, segmenting the image into a plurality of semantic regions; as well as Based on the edge and the plurality of semantic regions, an object region covering the object is identified.

2. The method according to claim 1, further comprising: According to the text detection model, a text area in the image is detected.

3. The method of claim 2, wherein identifying the object area comprises: Extracting edge features of the edge, semantic features of the plurality of semantic regions, and text features of the text region; Concatenating the edge feature, the semantic feature, and the text feature into a global feature; and Based on the global features, the object region is identified according to a trained classification model.

4. The method according to claim 3, further comprising: A morphological transformation is performed on the edges to make the edges continuous and closed.

5. The method according to claim 3, further comprising: Extracting an image frame of an input video, wherein the image frame includes the image; obtaining an indication from a user of a target object; as well as An object region in the image frame corresponding to the target object is identified.

6. The method according to claim 5, further comprising: Obtaining a replacement request for the target object, the replacement request including a replacement object; removing an object region of the target object from the image frame; as well as The object area is filled with a replacement object having a shape corresponding to the object area.

7. The method according to claim 5, further comprising: Acquire a training video, the training video comprising a set of image frames describing continuous motion of an object; identifying a first location of the object in a first image frame of the set of image frames; identifying a second location of the object in a second image frame of the set of image frames; determining a set of positions of the object in image frames between the first image frame and the second image frame based on the first position, the second position, and the continuous motion; annotating positions of the objects in the set of image frames based on the first position, the second position, and the set of positions to obtain a set of training data; as well as The classification model is trained using the set of training data.

8. The method of claim 1 , wherein identifying the edge comprises: Based on the values ​​of the pixels in the image, an edge in the image is detected according to a trained edge detection model, wherein a difference in the values ​​of the pixels on both sides of the edge is greater than a predetermined pixel threshold.

9. The method of claim 1, wherein segmenting the image into the plurality of semantic regions comprises: Based on the category of the object and according to a semantic segmentation model, the image is segmented into the multiple semantic regions.

10. An electronic device comprising: processor; as well as a volatile memory coupled to the processor, the volatile memory having instructions stored therein, the instructions, when executed by the processor, causing the device to perform actions, the actions comprising: Detecting edges in the image based on pixels in the image; Segmenting the image into a plurality of semantic regions based on the semantics of objects in the image; and Based on the edge and the plurality of semantic regions, an object region covering the object is identified.

11. The electronic device according to claim 10, wherein the actions further comprise: According to the text detection model, a text area in the image is detected.

12. The electronic device of claim 11, wherein identifying the object area comprises: Extracting edge features of the edge, semantic features of the plurality of semantic regions, and text features of the text region; Concatenating the edge feature, the semantic feature, and the text feature into a global feature; Based on the global features, the object region is identified according to a trained classification model.

13. The electronic device according to claim 12, wherein the actions further comprise: Extracting initial features of the edge; as well as Based on the initial features, the edges are denoised to make the edges continuous and closed.

14. The electronic device according to claim 12, wherein the actions further comprise: Extracting an image frame of an input video, wherein the image frame includes the image; obtaining an indication from a user of a target object; as well as An object region of the target object in the image frame is identified.

15. The electronic device according to claim 14, wherein the actions further comprise: Obtaining a replacement request for the target object, the replacement request including a replacement object; removing an object region of the target object from the image frame; as well as The object area is filled with a replacement object having a shape corresponding to the object area.

16. The electronic device according to claim 14, wherein the actions further comprise: Acquire a training video, the training video comprising a set of image frames describing continuous motion of an object; identifying a first location of the object in a first image frame of the set of image frames; identifying a second location of the object in a second image frame of the set of image frames; determining a set of positions of the object in image frames between the first image frame and the second image frame based on the first position, the second position, and the continuous operation; annotating positions of the objects in the set of image frames based on the first position, the second position and the set of positions to obtain a set of training data; as well as The classification model is trained using the set of training data.

17. The electronic device of claim 10, wherein identifying the edge comprises: Based on the values ​​of the pixels in the image, an edge in the image is detected according to a trained edge detection model, wherein a difference in the values ​​of the pixels on both sides of the edge is greater than a predetermined pixel threshold.

18. The electronic device of claim 10, wherein segmenting the image into the plurality of semantic regions comprises: Based on the category of the object and according to a semantic segmentation model, the image is segmented into the multiple semantic regions.

19. A computer program product tangibly stored on a computer readable medium and comprising machine executable instructions which, when executed, cause a machine to: Detecting edges in the image based on pixels in the image; Based on the semantics of objects in the image, segmenting the image into a plurality of semantic regions; as well as Based on the edge and the plurality of semantic regions, an object region covering the object is identified.

20. The computer program product of claim 19, wherein the machine executable instructions, when executed, further cause the machine to: According to the text detection model, a text area in the image is detected.