Image processing method and apparatus

Abstract

Embodiments of the present application disclose an image processing method and device; in the present application, a current frame image of a video is acquired, and the current frame image is identified; if the target object is identified on the current frame image, an initial position of the target object on the current frame image and an initial sub-position of the target object on a previous frame image of the current frame image are acquired; a change value is determined according to the initial position and the initial sub-position; the initial position is adjusted according to the change value and the initial sub-position, to obtain a target position of the target object on the current frame image. The present application can make the obtained target position more accurate. Embodiments of the present application can be applied to cloud technology, artificial intelligence, intelligent transportation, assisted driving and various scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image processing method and device. BACKGROUND

[0002] With the development of science and technology, computer vision is applied more and more widely. For example, computer vision is used in target object detection.

[0003] At present, when a target object is recognized on a current frame image, in order to prevent jitter phenomenon, the position of the target object on the current frame image is corrected. However, the accuracy of the current correction method is low. SUMMARY

[0004] The embodiments of the present application provide an image processing method and device, which can solve the technical problem of low accuracy of the current correction method.

[0005] An image processing method comprises:

[0006] obtaining a current frame image of a video, and recognizing the current frame image;

[0007] if the target object is recognized on the current frame image, obtaining an initial position of the target object on the current frame image and an initial sub-position of the target object on a previous frame image of the current frame image;

[0008] determining a change value according to the initial position and the initial sub-position;

[0009] adjusting the initial position according to the change value and the initial sub-position to obtain a target position of the target object on the current frame image.

[0010] Optionally, the adjusting the initial position according to the change value and the initial sub-position to obtain a target position of the target object on the current frame image comprises:

[0011] if the change value is greater than a first threshold value, the initial sub-position is taken as the target position of the target object on the current frame image.

[0012] Optionally, the adjusting the initial position according to the change value and the initial sub-position to obtain a target position of the target object on the current frame image comprises:

[0013] if the change value is less than or equal to a first threshold value, a motion state of the target object is determined.

[0014] If the motion state is a non-stationary state, the change value, the initial position and the initial sub-position are subjected to a non-linear operation to obtain a target position of the target object in the current frame image.

[0015] Optionally, the non-linear operation on the change value, the initial position and the initial sub-position to obtain the target position of the target object in the current frame image comprises:

[0016] determining a first weight corresponding to the initial position and a second weight corresponding to the initial sub-position based on a non-linear function and the change value;

[0017] determining the target position of the target object in the current frame image according to the first weight, the second weight, the initial position and the initial sub-position.

[0018] Optionally, the sum of the first weight and the second weight is 1, the first weight increases with the increase of the change value, and the second weight decreases with the increase of the change value.

[0019] Correspondingly, the determination of the first weight corresponding to the initial position and the second weight corresponding to the initial sub-position based on the non-linear function and the change value comprises:

[0020] substituting the change value into the non-linear function to obtain a function value;

[0021] If the function value increases with the increase of the change value, the function value is taken as the first weight corresponding to the initial position, and the second weight corresponding to the initial sub-position is determined based on the first weight;

[0022] If the function value decreases with the increase of the change value, the function value is taken as the second weight corresponding to the initial sub-position, and the first weight corresponding to the initial position is determined based on the second weight.

[0023] Optionally, the determination of the motion state of the target object if the change value is less than or equal to a first threshold value comprises:

[0024] If the change value is less than or equal to the first threshold value and greater than a second threshold value, the motion state of the target object is determined as a non-stationary state.

[0025] Optionally, after the identification of the current frame image, the method further comprises:

[0026] If the target object is not recognized on the current frame image, an initial historical position of the target object on a historical frame image is obtained, the historical frame image is an image before the current frame image in the video, and the historical frame image includes at least two frame images;

[0027] A motion track of the target object is determined according to the initial historical position;

[0028] A target position of the target object on the current frame image is estimated according to the motion track.

[0029] Optionally, after the target object is not recognized on the current frame image, the method further includes:

[0030] A frame number of historical frame images in which the target object is not detected continuously is determined;

[0031] Correspondingly, the initial historical position of the target object on the historical frame image is obtained, including:

[0032] If the frame number is less than or equal to a third threshold value, the initial historical position of the target object on the historical frame image is obtained.

[0033] Optionally, after the frame number of historical frame images in which the target object is not detected continuously is determined, the method further includes:

[0034] If the frame number is less than or equal to a third threshold value, an initial historical position of the target object on a target frame image is obtained, the target frame image being a last historical frame image in which the target object is detected;

[0035] The initial historical position on the target frame image is taken as a target position of the target object on the current frame image.

[0036] Optionally, after the target position of the target object on the current frame image is obtained, the method further includes:

[0037] The target object is displayed at the target position.

[0038] Correspondingly, an image processing device is provided, including:

[0039] A recognition module is configured to obtain a current frame image of a video and recognize the current frame image;

[0040] An obtaining module is configured to, if the target object is recognized on the current frame image, obtain an initial position of the target object on the current frame image and an initial sub-position of the target object on a previous frame image of the current frame image.

[0041] determining a change value according to the initial position and the initial sub-position;

[0042] adjusting the initial position according to the change value and the initial sub-position to obtain a target position of the target object in the current frame image.

[0043] In addition, an electronic device is further provided in the embodiment of the present application, which comprises a processor and a memory. The memory stores a computer program. The processor is used to run the computer program in the memory to realize the image processing method provided in the embodiment of the present application.

[0044] In addition, a computer readable storage medium is further provided in the embodiment of the present application. The computer readable storage medium stores a computer program. The computer program is suitable for being loaded by a processor to execute any one of the image processing methods provided in the embodiment of the present application.

[0045] In addition, a computer program product is further provided in the embodiment of the present application, which comprises a computer program. The computer program is executed by a processor to realize any one of the image processing methods provided in the embodiment of the present application.

[0046] In the embodiment of the present application, a current frame image of a video is acquired first, and the current frame image is identified. If a target object is identified in the current frame image, an initial position of the target object in the current frame image and an initial sub-position of the target object in a previous frame image of the current frame image are acquired. Then, a change value is determined according to the initial position and the initial sub-position. Finally, the initial position is adjusted according to the change value and the initial sub-position to obtain a target position of the target object in the current frame image.

[0047] That is, in the embodiment of the present application, the initial position of the target object in the current frame image at different time sequences and the initial sub-position of the previous frame image of the current frame image at different time sequences are different. Therefore, the initial position and the initial sub-position change, the change value is determined according to the initial position and the initial sub-position, and the change value also changes. When the initial position is adjusted according to the change value and the initial sub-position, the initial position can be adjusted based on the change of the initial position and the initial sub-position in real time, that is, the initial position can be adjusted based on the difference of the positions of the target object in the frame images at different time sequences, so that the obtained target position is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0049] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present application;

[0050] Figure 2 is a schematic diagram of a target object recognition method provided by an embodiment of the present application;

[0051] Figure 3 is a schematic diagram of another target object recognition method provided by an embodiment of the present application;

[0052] Figure 4 is a schematic diagram of a target position determination method provided by an embodiment of the present application;

[0053] Figure 5 is a schematic diagram of an image processing process provided by an embodiment of the present application;

[0054] Figure 6 is a schematic diagram of another target position determination method provided by an embodiment of the present application;

[0055] Figure 7 is a schematic diagram of another image processing method provided by an embodiment of the present application;

[0056] Figure 8 is a flowchart of another image processing method provided by an embodiment of the present application;

[0057] Figure 9 is a structural schematic diagram of an image processing apparatus provided by an embodiment of the present application;

[0058] Figure 10 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0060] The embodiments of the present application provide an image processing method, apparatus and computer readable storage medium. The image processing apparatus can be integrated in an electronic device, which can be a server, a terminal or other device.

[0061] The server can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0062] In addition, the plurality of servers can form a blockchain, and the servers are nodes on the blockchain.

[0063] The terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart home appliance, a vehicle terminal, and the like, but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0064] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, and assisted driving.

[0065] In addition, "plurality" in the embodiments of the present application refers to two or more. "First" and "second" and the like in the embodiments of the present application are used to distinguish the description, and cannot be understood as implying relative importance.

[0066] The following are described in detail respectively. It should be noted that the order of the following embodiments is not limited as the preferred order of the embodiments.

[0067] In the process of playing the video, when the target object is recognized on the current frame image, the position of the recognized target object on the current frame image can be wrong, at this time, the jitter phenomenon will also occur. Therefore, it is necessary to correct the position of the target object on the current frame image.

[0068] In the related art, the method for correcting the position of the target object on the current frame image is: setting a fixed weight value, and fusing the initial sub-position of the target object on the previous frame image of the current frame image and the initial position of the target object on the current frame image according to the fixed weight value, to obtain the target position of the target object on the current frame image.

[0069] However, because the change amplitude of the target object between images can be different. For example, the change amplitude of the position of the target object on the third frame image and the position of the target object on the fourth frame image in the video is different from the change amplitude of the position of the target object on the fourth frame image and the position of the target object on the fifth frame image in the video, so the use of a fixed weight coefficient makes the accuracy of the final target position not high.

[0070] To solve the technical problem, in the embodiment of the present application, a current frame image of a video is acquired first, and the current frame image is identified. If a target object is identified on the current frame image, an initial position of the target object on the current frame image and an initial sub-position of the target object on a previous frame image of the current frame image are acquired. Then, a change value is determined according to the initial position and the initial sub-position. Finally, the initial position is adjusted according to the change value and the initial sub-position to obtain a target position of the target object on the current frame image.

[0071] That is, in the embodiment of the present application, because the initial position of the target object on the current frame image at different time sequences and the initial sub-position of the current frame image at different time sequences are different, the initial position and the initial sub-position will change, the change value is determined according to the initial position and the initial sub-position, and the change value will also change. Then, when the initial position is adjusted according to the change value and the initial sub-position, the initial position can be adjusted based on the change of the initial position and the initial sub-position in real time, that is, the initial position can be adjusted based on the difference of the positions of the target object on the frame images at different time sequences, so that the obtained target position is more accurate.

[0072] In the embodiment, the image processing device is described from the perspective of the image processing device, which can be integrated in a server or a terminal or the like. For the convenience of describing the image processing method of the present application, the image processing device is integrated in a terminal, and the terminal is taken as an execution subject for detailed description.

[0073] Please refer to Figure 1 , Figure 1 is a flowchart of an image processing method provided by an embodiment of the present application. The image processing method can include:

[0074] S101, a current frame image of a video is acquired, and the current frame image is identified.

[0075] The current frame image refers to an image in the video to be processed by the terminal. The terminal can acquire the video first, and then acquire the current frame image of the video and identify the current frame image when receiving a video playing instruction, that is, the terminal can acquire the current frame image in the process of playing the video. Alternatively, the terminal can acquire the current frame image of the video and identify the current frame image when acquiring the video, that is, the terminal can acquire the current frame image before playing the video.

[0076] It should be understood that the terminal can obtain the video by shooting the video by the camera of the terminal itself. Alternatively, another terminal can shoot the video, and then the other terminal can send the video to the terminal, and the terminal can obtain the video. The user can select the specific manner of obtaining the video according to actual conditions, and the present application does not make any limitation here.

[0077] The user can select the specific manner of obtaining the video according to actual conditions, and the present application does not make any limitation here.

[0078] S102, if the target object is recognized on the current frame image, the initial position of the target object on the current frame image and the initial sub-position of the target object on the previous frame image of the current frame image are obtained.

[0079] The initial position on the current frame image refers to the coordinate of the target object on the current frame image. Similarly, the initial sub-position refers to the coordinate of the target object on the previous frame image of the current frame image. For example, the left lower corner pixel of the current frame image is taken as the coordinate axis origin to establish a coordinate system, and then the coordinate of the target object on the coordinate system can be obtained, and thus the coordinate of the target object on the current frame image can be obtained.

[0080] The target object is an object included in the video. The target object can be an entirety, for example, the target object can be a person or a dog. Alternatively, the target object can also be a target point. For example, the target object is a key point of a human body.

[0081] It should be understood that the target object can be one or multiple. After the terminal recognizes the current frame image, the first candidate object can be obtained. Then, the second candidate object obtained by recognizing the historical frame image of the current frame image is obtained. Then, the first candidate object and the second candidate object are compared. If there is a second candidate object different from the first candidate object, the second candidate object different from the first candidate object is taken as the target object, and it is determined that the target object is not recognized on the current frame image. If there is no second candidate object different from the first candidate object, the first candidate object is taken as the target object, and it is determined that the target object is recognized on the current frame image.

[0082] For example, as shown in FIG. 2, the target object is a person, and the target object is a key point of the person. Figure 2As shown in the figure, after recognizing the current frame image, first candidate object 1 and first candidate object 2 are obtained, and after recognizing the historical frame image of the current frame, second candidate object 3, second candidate object 4 and second candidate object 5 are obtained. There is second candidate object 5 which is different from first candidate object 1 and first candidate object 2. Second candidate object 5 is taken as the target object, and it is determined that the target object is not recognized in the current frame image.

[0083] For example, as shown in the figure, after recognizing the current frame image, first candidate object 1, first candidate object 2 and first candidate object 3 are obtained, and after recognizing the historical frame image of the current frame, second candidate object 4, second candidate object 5 and second candidate object 6 are obtained. There is no second candidate object which is different from first candidate object 1, first candidate object 2 and first candidate object 3, so it is determined that the target object is recognized in the current frame image. Figure 3

[0084] S103, determining a change value according to the initial position and the initial sub-position.

[0085] After obtaining the initial position and the initial sub-position, the terminal can determine a change value according to the initial position and the initial sub-position. The initial position and the initial sub-position can be subtracted to obtain the change value. Alternatively, the initial position and the initial sub-position can be divided to obtain the change value.

[0086] S104, adjusting the initial position according to the change value and the initial sub-position to obtain a target position of the target object on the current frame image.

[0087] After obtaining the change value, the terminal adjusts the initial position according to the change value and the initial sub-position to obtain a target position of the target object on the current frame image.

[0088] The process of adjusting the initial position according to the change value and the initial sub-position to obtain a target position of the target object on the current frame image can be as follows: Figure 4 If the change value is greater than a first threshold value, it indicates that the change range of the initial position and the initial sub-position is large, that is, the initial position is wrong, and at this time the initial sub-position is taken as the target position of the target object on the current frame image.

[0089] If the change value is less than or equal to the first threshold value, it indicates that the initial position has an error, and at this time the motion state of the target object is determined. If the motion state is a non-stationary state, the change value, the initial position and the initial sub-position are subjected to a nonlinear operation to obtain the target position of the target object on the current frame image.

[0090]

[0091] ​​If the motion state is a static state, the change range of the initial position and the initial sub-position is small, and the initial sub-position can be directly used as the target position of the target object in the current frame image.

[0092] The method for determining the motion state of the target object can be:

[0093] If the change value is less than or equal to the first threshold value and greater than the second threshold value, the motion state of the target object is determined as a non-static state. If the change value is less than or equal to the first threshold value and less than or equal to the second threshold value, the motion state of the target object is determined as a static state.

[0094] Alternatively, when the target object is static and the terminal recording the video is in motion, the motion state of the target object can also be determined through a sensor on the terminal.

[0095] The first threshold value and the second threshold value are preset values. The user can set them according to the actual situation, which is not limited in the present application.

[0096] In some embodiments, the change value, the initial position and the initial sub-position are subjected to nonlinear operation to obtain the target position of the target object in the current frame image, including:

[0097] The first weight corresponding to the initial position and the second weight corresponding to the initial sub-position are determined based on the nonlinear function and the change value;

[0098] The target position of the target object in the current frame image is determined according to the first weight, the second weight, the initial position and the initial sub-position.

[0099] In the present embodiment, the first weight corresponding to the initial position and the second weight corresponding to the initial sub-position are determined based on the nonlinear function and the change value, so that the first weight and the second weight change nonlinearly with the change of the change value, and the target position of the target object in the current frame image obtained according to the first weight, the second weight, the initial position and the initial sub-position is more accurate.

[0100] The specific type of the nonlinear function can be selected according to the actual situation, such as selecting a cosine function or a logarithmic function as the nonlinear function in the present embodiment, which is not limited in the present application.

[0101] Optionally, the sum of the first weight and the second weight is 1, the first weight becomes larger as the change value becomes larger, and the second weight becomes smaller as the change value becomes larger.

[0102] Correspondingly, the first weight corresponding to the initial position and the second weight corresponding to the initial sub-position are determined based on the nonlinear function and the change value, including:

[0103] The change value is substituted into the nonlinear function to obtain a function value;

[0104] If the function value increases as the change value increases, the function value is taken as a first weight corresponding to the initial position, and a second weight corresponding to the initial sub-position is determined based on the first weight;

[0105] If the function value decreases as the change value increases, the function value is taken as a second weight corresponding to the initial sub-position, and a first weight corresponding to the initial position is determined based on the second weight.

[0106] For example, when the function value decreases as the change value increases (for example, when the nonlinear function is a cosine function, the function value decreases as the change value increases), the change value, the initial position, and the initial sub-position can be substituted into the following formula to obtain the target position:

[0107] l ti =pl ti f(d ti )+ml ti (1-f(d ti ))

[0108] wherein l ti represents the target position, pl ti represents the initial sub-position, ml ti represents the initial position, d ti represents the change value, f() represents the nonlinear function, f(d ti ) represents the second weight, (1-f(d ti )) represents the first weight, t represents the time sequence, and i represents the target object.

[0109] In this embodiment, when the target object is in a non-stationary state and the change value is large, it indicates that the target object changes quickly, that is, the difference between the target position on the current frame image and the initial position is small, and the difference between the target position on the current frame image and the initial sub-position is large, so the first weight of the initial position can be increased and the second weight of the initial sub-position can be decreased.

[0110] When the target object is in a non-stationary state and the change value is small, it indicates that the target object changes slowly, that is, the difference between the target position on the current frame image and the initial sub-position is small, and the difference between the target position on the current frame image and the initial position is large, so the second weight of the initial sub-position can be increased and the first weight of the initial position can be decreased.

[0111] Therefore, the sum of the first weight and the second weight is 1, the first weight increases as the change value increases, and the second weight decreases as the change value increases, so that the finally obtained target position can be more accurate.

[0112] In this embodiment, the current frame of the video is first acquired and then identified. If a target object is identified in the current frame, the initial position of the target object in the current frame and its initial sub-position in the previous frame are obtained. Then, a change value is determined based on the initial position and the initial sub-position. Finally, the initial position is adjusted based on the change value and the initial sub-position to obtain the target position of the target object in the current frame.

[0113] In this embodiment, since the initial position of the target object in the current frame image at different time sequences is different from the initial sub-position in the previous frame image of the current frame image at different time sequences, the initial position and the initial sub-position will change. The change value is determined based on the initial position and the initial sub-position, and the change value will also change. Therefore, when adjusting the initial position based on the change value and the initial sub-position, the initial position can be adjusted in real time based on the changes in the initial position and the initial sub-position. That is, the initial position can be adjusted based on the difference in the position of the target object in the frame images at different time sequences, thereby making the obtained target position more accurate.

[0114] During video playback, if the target object is not detected in the current frame of the video, the terminal will not display the target object in the current frame. When the terminal detects the target object in the next frame, the terminal will display the target object in the next frame. At this time, the user will feel that the target object is jittering.

[0115] For example, such as Figure 5 As shown, if the target object is detected in the previous frame of the current frame of the video, but not in the current frame, and then detected in the next frame, meaning the target object was missed in the current frame, the user will feel that the target object is jittering after playing from the current frame to the next frame.

[0116] To address the aforementioned technical problems, in some other embodiments, this application further includes:

[0117] If no target object is identified in the current frame, the initial historical position of the target object in the historical frame is obtained. The historical frame is the image before the current frame in the video, and the historical frame includes at least two frames.

[0118] Determine the movement trajectory of the target object based on its initial historical location;

[0119] Estimate the target position of the target object in the current frame image based on the motion trajectory.

[0120] If the terminal does not identify the target object on the current frame image after identifying the current frame image, the terminal acquires an initial historical position of the target object on a historical frame image, the historical frame image being an image before the current frame image in the video.

[0121] The initial historical position refers to a coordinate of the target object on the historical frame image. For example, a coordinate system is established with a left-bottom pixel of the historical frame image as an origin of a coordinate axis, and a coordinate of the target object on the coordinate system can be obtained, so that a coordinate of the target object on the historical frame image can be obtained.

[0122] For the method of determining the motion trajectory of the target object, a user can select according to actual conditions, for example, in an embodiment of the present application, a deep learning algorithm or a parameter method is selected to determine the motion trajectory of the target object, which is not limited in the present application.

[0123] In the embodiment, a current frame image of a video is acquired first, and the current frame image is identified. If the target object is not identified on the current frame image, an initial historical position of the target object on a historical frame image is acquired, the historical frame image being an image before the current frame image in the video. Then, the motion trajectory of the target object is determined according to the initial historical position. Finally, the target position of the target object on the current frame image is estimated according to the motion trajectory.

[0124] That is, in the embodiment of the present application, when the target object is not identified on the current frame image, the motion trajectory of the target object is calculated according to the initial positions of the target object on at least two historical frame images, and then the target position of the target object on the current frame image is determined according to the motion trajectory of the target object, so that shaking caused by missing the target object on the current frame image can be avoided.

[0125] When the target object is not identified on the current frame image and the target object is identified on the previous N historical frame images (N can be set according to actual conditions), it is indicated that the target object is lost. Since when the target object is lost to a high degree, the accuracy of the target position of the target object on the current frame image obtained according to the initial historical position is not high. Therefore, in some embodiments, when the target object is not identified on the current frame image, the loss degree of the target object can also be determined. Optionally, the process of determining the loss degree of the target object can be as follows:

[0126] The frame number of the historical frame images on which the target object is not detected continuously is determined.

[0127] Correspondingly, the initial historical position of the target object on the historical frame image includes:

[0128] If the frame number is less than or equal to the third threshold value, it is indicated that the loss degree of the target object is not high, and the initial historical position of the target object on the historical frame image can be acquired.

[0129] The number of consecutive history frame images in which the target object is not detected can be, for example, if the current frame image is the tenth frame image in a video, the target object is not detected on the fifth frame image and the ninth frame in the video, and the number of consecutive history frame images in which the target object is not detected is five.

[0130] The process of determining the number of consecutive history frame images in which the target object is not detected can be determining the identification state of the target object on each history frame image. If the identification state is the no identification state, the number of history frame images corresponding to the no identification state is determined. The identification state includes the no identification state and the identification state. When the target object is identified on a history frame image, the identification state of the history frame image is the identification state, and when the target object is not identified on a history frame image, the identification state of the history frame image is the no identification state.

[0131] In this embodiment, the loss degree of the target object is determined by the number of consecutive history frame images in which the target object is not detected. When the number of consecutive history frame images in which the target object is not detected is less than or equal to the third threshold value, the initial history position of the target object on the history frame image is obtained again, thereby improving the accuracy of the target position.

[0132] The third threshold value is a pre-set value. The user can set it according to the actual situation, which is not limited herein.

[0133] If the target object is not detected on the current frame image and the target object is not detected on the previous N history frame images, it means that the target object has never appeared, and at this time, it is determined that the target object is not detected on the current frame image.

[0134] In other embodiments, since the time interval between different frame images is relatively short, if the number of consecutive history frame images in which the target object is not detected is less than or equal to the third threshold value, only the initial history position of the target object on the target frame image can be obtained, the target frame image is the last history frame image in which the target object is detected, and then the initial history position on the target frame image is taken as the target position of the target object on the current frame image, that is, the initial history position of the target frame image can be directly reused at this time (see Figure 6 ).

[0135] For example, if the current frame image is the tenth frame image in a video, the target object is not detected on the fifth frame image and the ninth frame in the video, the initial history position of the target object on the fourth history frame image can be taken as the target position of the target object on the current frame image.

[0136] If the number of historical frame images in which the target object is not detected consecutively is less than or equal to the third threshold value, it is determined that the target object is not identified on the current frame image.

[0137] After obtaining the target position of the target object on the current frame image, the target object is displayed at the target position, so that the user can see the target object on the current frame image.

[0138] As can be seen from the above, in the embodiments of the present application, the method for solving the technical problem of target object jitter includes two kinds, one is the image processing method executed when the target object is not identified as described above, and the other is the image processing method executed when the target object is identified as described above. Therefore, in the embodiments of the present application, first, the different time sequence frame images (such as, for example, frame image (t) of the video, frame image (t-1) of the video, etc.) in the video are identified, then the object relationship between the objects of different frame images is determined, and then it is judged whether there is a target object, if there is a target object, the image processing method executed when the target object is identified is executed, if there is no target object, the image processing method executed when the target object is not identified is executed. Figure 7

[0139] According to the method described in the above embodiments, the following will be further described in detail by way of example.

[0140] This embodiment takes the image processing device integrated in a terminal as an example, please refer to Figure 8 , Figure 8 The flowchart of the image processing method provided in the embodiments of the present application. The image processing method flow can include:

[0141] S801, the terminal acquires the current frame image of the video, and identifies the current frame image.

[0142] The terminal shoots a video through its own camera, and then acquires the current frame image of the video when receiving a video playing instruction, and identifies the current frame image through a convolution network model.

[0143] S802, if the target object is not identified on the current frame image, the terminal determines the number of historical frame images in which the target object is not detected consecutively, and the historical frame image is the image before the current frame image in the video.

[0144] When the target object is not identified on the current frame image, it means that the target object is lost. Since when the target object is lost to a serious degree, the accuracy of the target position of the target object on the current frame image obtained at last is not high.

[0145] ​In order to obtain the target position more accurately, if the target object is not recognized on the current frame image, the number of historical frame images in which the target object is not detected continuously is determined.

[0146] The number of historical frame images in which the target object is not detected continuously may be, for example, that the current frame image is the tenth frame image in a video, the target object is not recognized on the fifth frame image and the ninth frame image in the video, and the number of historical frame images in which the target object is not detected continuously is five.

[0147] The process of determining the number of historical frame images in which the target object is not detected continuously may be that the recognition state of the target object on each historical frame image is determined. If the recognition state is the state of not being recognized, the number of historical frame images corresponding to the state of not being recognized is determined. The recognition state includes the state of not being recognized and the state of being recognized. When the target object is recognized on a historical frame image, the recognition state of the historical frame image is the state of being recognized, and when the target object is not recognized on a historical frame image, the recognition state of the historical frame image is the state of not being recognized.

[0148] The target object is an object included in the video. The target object may be a whole, for example, the target object may be a person or a dog. Alternatively, the target object may also be a target point. For example, the target object is a key point of a human body.

[0149] It should be understood that the target object may be one or more. After the terminal recognizes the current frame image, a first candidate object is obtained. Then, a second candidate object obtained by recognizing the historical frame image is obtained. Then, the first candidate object and the second candidate object are compared. If there is a second candidate object different from the first candidate object, the second candidate object different from the first candidate object is taken as the target object, and it is determined that the target object is not recognized on the current frame image. If there is no second candidate object different from the first candidate object, the first candidate object is taken as the target object, and it is determined that the target object is recognized on the current frame image.

[0150] S803, if the number of frames is less than or equal to the third threshold value, the terminal obtains an initial historical position of the target object on the historical frame image, and the historical frame image includes at least two frame images.

[0151] After determining the number of historical frame images in which the target object is not detected continuously, if the number of frames is less than or equal to the third threshold value, it indicates that the loss degree of the target object is not high, and the initial historical position of the target object on the historical frame image is obtained.

[0152] The initial historical position refers to a coordinate of the target object on a historical frame image. For example, a coordinate system is established with a left-bottom pixel of the historical frame image as an origin of a coordinate axis, and a coordinate of the target object on the coordinate system can be obtained, so that a coordinate of the target object on the historical frame image can be obtained.

[0153] The third threshold value is a preset value. The user can set the value according to an actual situation, which is not limited in this application.

[0154] S804, the terminal determines a motion track of the target object according to the initial historical position.

[0155] After the terminal obtains the initial historical position of the target object on each historical frame image, the terminal can determine the motion track of the target object according to each initial historical position.

[0156] S805, the terminal estimates a target position of the target object on the current frame image according to the motion track.

[0157] After the terminal obtains the motion track of the target object, the terminal estimates the target position of the target object on the current frame image according to the motion track, and displays the target object at the target position, so that the current frame image includes the target object, and thus, shaking caused by missing the target object in the current frame image is avoided.

[0158] In the embodiment of the application, when the target object is not recognized on the current frame image, the target position of the target object on the current frame image is determined according to the motion track of the target object, so that shaking caused by missing the target object in the current frame image is avoided.

[0159] S806, if the target object is recognized on the current frame image, the terminal obtains an initial position of the target object on the current frame image and an initial sub-position of the target object on a previous frame image of the current frame image, and determines a change value according to the initial position and the initial sub-position.

[0160] When the target object is recognized on the current frame image, the position of the recognized target object on the current frame image can be incorrect, and in this case, shaking phenomenon occurs. Therefore, the position of the target object on the current frame image needs to be adjusted.

[0161] Therefore, if the target object is recognized on the current frame image, the initial position of the target object on the current frame image and the initial sub-position of the target object on the previous frame image of the current frame image are obtained.

[0162] The initial position and the initial sub-position can be subtracted to obtain the change value. Alternatively, the initial position and the initial sub-position can be divided to obtain the change value.

[0163] S807, if the change value is greater than the first threshold value, the terminal takes the initial sub-position as the target position of the target object on the current frame image.

[0164] If the change value is greater than the first threshold value, it means that the change range of the initial position and the initial sub-position is large, that is, the initial position is wrong, at this time, the initial sub-position is directly taken as the target position of the target object on the current frame image.

[0165] S808, if the change value is less than or equal to the first threshold value, the terminal determines the motion state of the target object.

[0166] When the target object is stationary, and the terminal recording the video is in motion, the motion state of the target object can be determined through the sensor on the terminal.

[0167] Alternatively, the motion state of the target object can also be determined as follows: if the change value is less than or equal to the first threshold value, and the change value is greater than the second threshold value, the motion state of the target object is determined as a non-stationary state. If the change value is less than or equal to the first threshold value, and the change value is less than or equal to the second threshold value, the motion state of the target object is determined as a non-stationary state.

[0168] S809, if the motion state is a stationary state, the terminal takes the initial sub-position as the target position of the target object on the current frame image.

[0169] S8010, if the motion state is a non-stationary state, the terminal substitutes the change value into a non-linear function to obtain a function value.

[0170] S8011, if the function value increases as the change value increases, the terminal takes the function value as a first weight corresponding to the initial position, and determines a second weight corresponding to the initial sub-position based on the first weight, the sum of the first weight and the second weight is 1, the first weight increases as the change value increases, and the second weight decreases as the change value increases.

[0171] After obtaining the first weight, 1 is subtracted from the first weight to obtain the second weight corresponding to the initial sub-position.

[0172] S8012, if the function value decreases as the change value increases, the terminal takes the function value as a second weight corresponding to the initial sub-position, and determines a first weight corresponding to the initial position based on the second weight.

[0173] After obtaining the second weight, 1 is subtracted from the second weight to obtain the first weight corresponding to the initial position.

[0174] S8013, the target position of the target object on the current frame image is determined according to the first weight, the second weight, the initial position and the initial sub-position.

[0175] In this embodiment, since the initial position of the target object in the current frame image at different time sequences is different from the initial sub-position in the previous frame image of the current frame image at different time sequences, the initial position and the initial sub-position will change. The change value is determined based on the initial position and the initial sub-position, and the change value will also change. Therefore, a first weight and a second weight are determined based on a nonlinear function and the change value. Finally, the target position of the target object in the current frame image is determined based on the first weight, the second weight, the initial position, and the initial sub-position. This allows the initial position to be adjusted in real time based on the changes in the initial position and the initial sub-position, that is, the initial position can be adjusted based on the difference in the position of the target object in the frame images at different time sequences, thereby making the obtained target position more accurate.

[0176] In addition, when the target object is in a non-static state and the change value is large, it means that the target object is changing rapidly. That is, the difference between the target position and the initial position in the current frame image is small, while the difference between the target position and the initial sub-position in the current frame image is large. Therefore, the first weight of the initial position can be increased and the second weight of the initial sub-position can be decreased.

[0177] When the target object is in a non-static state and the change value is small, it means that the change of the target object is slow. That is, it means that the difference between the target position and the initial sub-position of the target object in the current frame image is small, and the difference between the target position and the initial position in the current frame image is large. Therefore, the second weight of the initial sub-position can be increased and the first weight of the initial position can be decreased.

[0178] Therefore, the sum of the first weight and the second weight is 1. The first weight increases as the change value increases, while the second weight decreases as the change value increases, which makes the final target position more accurate.

[0179] S8014. Display the target object at the target location.

[0180] To facilitate better implementation of the image processing method provided in the embodiments of this application, the embodiments of this application also provide an apparatus based on the above-described image processing method. The meanings of the terms used are the same as in the image processing method described above, and specific implementation details can be found in the descriptions in the method embodiments.

[0181] The image processing device can be integrated into electronic devices, such as servers or terminals, which may include smartphones, tablets, laptops, and / or personal computers.

[0182] For example, such as Figure 9 As shown, the image processing apparatus may include:

[0183] The recognition module 901 is configured to acquire a current frame image of the video and recognize the current frame image.

[0184] The acquisition module 902 is configured to, if the target object is recognized on the current frame image, acquire an initial position of the target object on the current frame image and an initial sub-position of the target object on a previous frame image of the current frame image.

[0185] The determination module 903 is configured to determine a change value according to the initial position and the initial sub-position.

[0186] The adjustment module 904 is configured to adjust the initial position according to the change value and the initial sub-position to obtain a target position of the target object on the current frame image.

[0187] Optionally, the adjustment module 904 is specifically configured to perform:

[0188] If the change value is greater than a first threshold value, the initial sub-position is taken as the target position of the target object on the current frame image.

[0189] Optionally, the adjustment module 904 is specifically configured to perform:

[0190] If the change value is less than or equal to the first threshold value, a motion state of the target object is determined.

[0191] If the motion state is a non-stationary state, the change value, the initial position and the initial sub-position are subjected to a non-linear operation to obtain the target position of the target object on the current frame image.

[0192] Optionally, the adjustment module 904 is specifically configured to perform:

[0193] A first weight corresponding to the initial position and a second weight corresponding to the initial sub-position are determined based on a non-linear function and the change value.

[0194] The target position of the target object on the current frame image is determined according to the first weight, the second weight, the initial position and the initial sub-position.

[0195] Optionally, a sum of the first weight and the second weight is 1, the first weight becomes larger as the change value becomes larger, and the second weight becomes smaller as the change value becomes larger.

[0196] Correspondingly, the adjustment module 904 is specifically configured to perform:

[0197] The change value is substituted into the non-linear function to obtain a function value;

[0198] If the function value becomes larger as the change value becomes larger, the function value is taken as the first weight corresponding to the initial position, and the second weight corresponding to the initial sub-position is determined based on the first weight.

[0199] If the function value decreases as the change value increases, the function value is taken as a second weight corresponding to the initial sub-position, and the first weight corresponding to the initial position is determined based on the second weight.

[0200] Optionally, the adjusting module 904 is specifically configured to perform:

[0201] If the change value is less than or equal to the first threshold value and greater than the second threshold value, the motion state of the target object is determined as a non-stationary state.

[0202] Optionally, the image processing apparatus further comprises:

[0203] The position obtaining module is configured to, if the target object is not recognized on the current frame image, obtain an initial historical position of the target object on a historical frame image, the historical frame image being an image before the current frame image in the video, and the historical frame image including at least two frame images.

[0204] The trajectory determining module is configured to determine a motion trajectory of the target object according to the initial historical position.

[0205] The estimating module is configured to estimate a target position of the target object on the current frame image according to the motion trajectory.

[0206] Optionally, the image processing apparatus further comprises:

[0207] The frame number determining module is configured to determine a frame number of the historical frame images on which the target object is not detected continuously.

[0208] Correspondingly, the position obtaining module is specifically configured to perform:

[0209] If the frame number is less than or equal to a third threshold value, the initial historical position of the target object on the historical frame image is obtained.

[0210] Optionally, the image processing apparatus further comprises:

[0211] The sub-position obtaining module is configured to, if the frame number is less than or equal to the third threshold value, obtain an initial historical position of the target object on a target frame image, the target frame image being the last historical frame image on which the target object is detected.

[0212] As a module, the initial historical position on the target frame image is taken as the target position of the target object on the current frame image.

[0213] Optionally, the image processing apparatus further comprises:

[0214] The display module is configured to display the target object at the target position.

[0215] In practice, the above various modules can be implemented as independent entities, or combined as the same or several entities, and the specific implementation manners of the above various modules and the corresponding beneficial effects can be referred to the method embodiments, which will not be described here.

[0216] The embodiments of the present application also provide an electronic device, which can be a server or a terminal, etc. Figure 10 As shown in the figure, it shows a structural schematic diagram of an electronic device related to the embodiments of the present application, in particular:

[0217] The electronic device can include a processor 1001 with one or more processing cores, a memory 1002 with one or more computer readable storage media, a power supply 1003, an input unit 1004, and the like. Those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements. Among them: Figure 10

[0218] The processor 1001 is the control center of the electronic device, which connects various parts of the entire electronic device through various interfaces and lines, and performs various functions and processes data of the electronic device by running or executing computer programs and / or modules stored in the memory 1002, and calling data stored in the memory 1002, thereby overall detecting the electronic device. Optionally, the processor 1001 can include one or more processing cores; preferably, the processor 1001 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 1001.

[0219] The memory 1002 can be used to store computer programs and modules, and the processor 1001 executes various function applications and data processing by running the computer programs and modules stored in the memory 1002. The memory 1002 can mainly include a program storage area and a data storage area, wherein the program storage area can store the operating system, at least one computer program required by the function (such as sound playing function, image playing function, etc.), etc.; the data storage area can store the data created according to the use of the electronic device, etc. In addition, the memory 1002 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 1002 can also include a memory controller to provide the processor 1001 with access to the memory 1002.

[0220] ​The electronic device can further include a power supply 1003 for supplying power to each component. Preferably, the power supply 1003 can be logically connected to the processor 1001 through a power management system, so that the power management system can realize functions such as management of charging, discharging, and power consumption management. The power supply 1003 can further include one or more than one direct current or alternating current power supply, a recharging system, a power supply fault detection circuit, a power supply converter or inverter, a power supply state indicator, and the like.

[0221] The electronic device can further include an input unit 1004 for receiving input digital or character information, and generating keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0222] Although not shown, the electronic device can further include a display unit and the like, which will not be described here. In the embodiment, the processor 1001 in the electronic device loads one or more than one executable file corresponding to the processes of the computer program into the memory 1002 according to the following instructions, and runs the computer program stored in the memory 1002 by the processor 1001, so as to realize various functions, such as:

[0223] Obtaining a current frame image of the video, and identifying the current frame image;

[0224] If a target object is identified on the current frame image, obtaining an initial position of the target object on the current frame image and an initial sub-position of the target object on a previous frame image of the current frame image;

[0225] Determining a change value according to the initial position and the initial sub-position;

[0226] Adjusting the initial position according to the change value and the initial sub-position to obtain a target position of the target object on the current frame image.

[0227] The specific implementation and corresponding beneficial effects of each operation above can be referred to the detailed description of the image processing method above, which will not be described here.

[0228] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a computer program, or by a computer program controlling related hardware, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0229] Therefore, an embodiment of the present application provides a computer readable storage medium, which stores a computer program capable of being loaded by a processor to execute steps in any image processing method provided by the embodiments of the present application. For example, the computer program can execute the following steps:

[0230] obtaining a current frame image of the video, and performing recognition on the current frame image;

[0231] if a target object is recognized on the current frame image, obtaining an initial position of the target object on the current frame image and an initial sub-position of the target object on a previous frame image of the current frame image;

[0232] determining a change value according to the initial position and the initial sub-position;

[0233] adjusting the initial position according to the change value and the initial sub-position to obtain a target position of the target object on the current frame image.

[0234] The specific implementation of each operation and the corresponding beneficial effects can be referred to the foregoing embodiments, which will not be repeated here.

[0235] The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0236] Due to the computer program stored in the computer readable storage medium, the steps of any of the image processing methods provided by the embodiments of the present application can be executed, and thus the beneficial effects of any of the image processing methods provided by the embodiments of the present application can be achieved. Details can be referred to the foregoing embodiments, which will not be repeated here.

[0237] According to an aspect of the present application, a computer program product or a computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device perform the image processing method.

[0238] The above describes in detail the image processing method, device and computer readable storage medium provided by the embodiments of the present application. The principle and implementation manner of the present application are described by applying specific examples in this paper. The above description of the embodiments is only used to help understand the method and its core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. An image processing method, characterized by, The method comprises the following steps: acquiring a current frame image of a video and performing recognition on the current frame image; if a target object is recognized on the current frame image, acquiring an initial position of the target object on the current frame image and an initial sub-position of the target object on a previous frame image of the current frame image; determining a change value according to the initial position and the initial sub-position; adjusting the initial position according to the change value and the initial sub-position to obtain a target position of the target object on the current frame image, comprising: if the change value is less than or equal to a first threshold value, determining a motion state of the target object; if the motion state is a non-stationary state, substituting the change value into a non-linear function to obtain a function value; if the function value increases as the change value increases, taking the function value as a first weight corresponding to the initial position, and determining a second weight corresponding to the initial sub-position based on the first weight; if the function value decreases as the change value increases, taking the function value as a second weight corresponding to the initial sub-position, and determining a first weight corresponding to the initial position based on the second weight; determining the target position of the target object on the current frame image according to the first weight, the second weight, the initial position and the initial sub-position, and obtaining the target position through the following formula: wherein, l ti represents a target position, pl ti represents an initial sub-position, ml ti represents an initial position, d ti represents a change value, f（） represents a non-linear function, f(d ti ） represents a second weight, (1- f(d ti ） ) represents a first weight, t represents a timing, i represents a target object.

2. The image processing method of claim 1, wherein, The adjusting of the initial position according to the change value and the initial sub-position to obtain the target position of the target object on the current frame image comprises: if the change value is greater than the first threshold value, taking the initial sub-position as the target position of the target object on the current frame image.

3. The image processing method of claim 1, wherein, The sum of the first weight and the second weight is 1, the first weight increases as the change value increases, and the second weight decreases as the change value increases.

4. The image processing method of claim 1, wherein, If the change value is less than or equal to a second threshold value, the motion state of the target object is determined, comprising: if the change value is less than or equal to the first threshold value and greater than the second threshold value, the motion state of the target object is determined as a non-stationary state.

5. The image processing method of claim 1, wherein, After the recognition on the current frame image, the method further comprises the following steps: if no target object is recognized on the current frame image, acquiring an initial historical position of the target object on a historical frame image, the historical frame image being an image before the current frame image in the video, and the historical frame image comprising at least two frame images; determining a motion trajectory of the target object according to the initial historical position; estimating a target position of the target object on the current frame image according to the motion trajectory.

6. The image processing method of claim 5, wherein, After the recognition on the current frame image, the method further comprises the following steps: determining a frame number of historical frame images in which the target object is not detected continuously; Accordingly, the acquiring of the initial historical position of the target object on the historical frame image comprises: if the frame number is less than or equal to a third threshold value, acquiring the initial historical position of the target object on the historical frame image.

7. The image processing method of claim 6, wherein, After the frame number of the historical frame images in which the target object is not continuously detected is determined, the method further comprises: if the frame number is less than or equal to a third threshold value, obtaining an initial historical position of the target object on a target frame image, the target frame image being a last historical frame image in which the target object is detected; taking the initial historical position on the target frame image as a target position of the target object on the current frame image.

8. The image processing method of any of claims 1-7, wherein, After the target position of the target object on the current frame image is obtained, the method further comprises: displaying the target object at the target position.

9. An image processing apparatus characterized by comprising: The method comprises: a recognition module, configured to obtain a current frame image of a video and recognize the current frame image; a obtaining module, configured to, if a target object is recognized on the current frame image, obtain an initial position of the target object on the current frame image and an initial sub-position of the target object on a previous frame image of the current frame image; a determining module, configured to determine a change value according to the initial position and the initial sub-position; an adjusting module, configured to adjust the initial position according to the change value and the initial sub-position to obtain a target position of the target object on the current frame image, comprising: if the change value is less than or equal to a first threshold value, determining a motion state of the target object; if the motion state is a non-stationary state, substituting the change value into a non-linear function to obtain a function value; if the function value increases as the change value increases, taking the function value as a first weight corresponding to the initial position, and determining a second weight corresponding to the initial sub-position based on the first weight; if the function value decreases as the change value increases, taking the function value as a second weight corresponding to the initial sub-position, and determining a first weight corresponding to the initial position based on the second weight; determining the target position of the target object on the current frame image according to the first weight, the second weight, the initial position and the initial sub-position, and obtaining the target position through the following formula: wherein, l ti represents a target position, pl ti represents an initial sub-position, ml ti represents an initial position, d ti represents a change value, f（） represents a non-linear function, f(d ti ） represents a second weight, (1- f(d ti ） ) represents a first weight, t represents a timing, i represents a target object.

Images