Image recognition method, device, electronic device and storage medium

Abstract

The present application discloses an image recognition method, device, electronic device and storage medium, which obtains images to be processed of a group to be detected, wherein the group to be detected includes images to be processed respectively acquired by multiple acquisition channels at the same acquisition time, then performs image recognition processing on the images to be processed of the group to be detected to obtain target object recognition results, and performs image stitching processing on the images to be processed of the group to be detected to obtain panoramic video frames, and then associates the panoramic video frames corresponding to the group to be detected with the target object recognition results. By performing image recognition processing on the acquired images to be processed and then associating the image recognition results with the corresponding panoramic video frames, the image recognition difficulty of the panoramic video frames can be simplified and the accuracy of image recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to an image recognition method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of virtual reality (VR) technology, panoramic videos are increasingly used in various fields, such as VR video live broadcast, VR games, VR simulations, etc. Each frame in a panoramic video is a rendered panoramic image, giving users an immersive feeling.

[0003] However, due to different rendering methods, the rendered video frames have technical difficulties such as distortion and complex calculations, making it difficult to perform further image processing and application. Summary of the invention

[0004] In view of the above problems, the present invention proposes an image recognition method, device, electronic device and storage medium to improve the above problems.

[0005] In a first aspect, an embodiment of the present application provides an image recognition method, which includes: obtaining images to be processed of a group to be detected, wherein the group to be detected includes images to be processed that are respectively acquired by multiple acquisition channels at the same acquisition time; performing image recognition processing on the images to be processed of the group to be detected to obtain target object recognition results; performing image stitching processing on the images to be processed of the group to be detected to obtain panoramic video frames; and associating the panoramic video frames corresponding to the group to be detected with the target object recognition results.

[0006] In the second aspect, an embodiment of the present application also provides an image recognition device, which includes: an image acquisition module, an image recognition module, an image stitching module and an association module. Among them, the image acquisition module is used to acquire the to-be-processed images of the to-be-detected group, wherein the to-be-detected group includes the to-be-processed images acquired by multiple acquisition channels at the same acquisition time. The image recognition module is used to perform image recognition processing on the to-be-detected images of the to-be-detected group to obtain target object recognition results. The image stitching module is used to perform image stitching processing on the to-be-detected images of the to-be-detected group to obtain panoramic video frames. The association module is used to associate the panoramic video frames corresponding to the to-be-detected group with the target object recognition results.

[0007] In a third aspect, an embodiment of the present application further provides an electronic device, the electronic device comprising: one or more processors; a memory; and one or more applications. The one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the image recognition method as described in the first aspect above.

[0008] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the image recognition method as described in the first aspect above.

[0009] The technical solution provided by the present invention obtains images to be processed of a group to be detected, wherein the group to be detected includes images to be processed that are respectively acquired by multiple acquisition channels at the same acquisition time, and then performs image recognition processing on the images to be processed of the group to be detected to obtain target object recognition results, and performs image stitching processing on the images to be processed of the group to be detected to obtain panoramic video frames, and then associates the panoramic video frames corresponding to the group to be detected with the target object recognition results. By performing image recognition processing on the acquired images to be processed and then associating the image recognition results with the corresponding panoramic video frames, the image recognition difficulty of the panoramic video frames can be simplified and the accuracy of image recognition can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments and drawings obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0011] Figure 1 A schematic diagram of an application environment involved in an embodiment of the present application is shown.

[0012] Figure 2 A schematic flow chart of an image recognition method proposed in an embodiment of the present application is shown.

[0013] Figure 3 A schematic diagram of a flow chart of performing image recognition processing in a sorted order proposed in an embodiment of the present application is shown.

[0014] Figure 4 A schematic diagram of a scenario for determining a moving area of ​​a target object proposed in an embodiment of the present application is shown.

[0015] Figure 5 A schematic diagram of another scenario for determining a moving area of ​​a target object proposed in an embodiment of the present application is shown.

[0016] Figure 6 A schematic diagram of the flow of image recognition processing when a target object is detected in multiple images to be processed is shown.

[0017] Figure 7 A schematic diagram of the flow of image recognition processing by combining images is shown.

[0018] Figure 8 A schematic diagram of a scenario for interacting with a host proposed in an embodiment of the present application is shown.

[0019] Fig. 9 A schematic diagram of another scenario of interacting with a host proposed in an embodiment of the present application is shown.

[0020] Fig.10 A structural block diagram of an image recognition device proposed in an embodiment of the present application is shown;

[0021] Fig.11 A structural block diagram of an electronic device proposed in an embodiment of the present application is shown;

[0022] Fig.12 A structural block diagram of a computer-readable storage medium proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0024] With the continuous development of virtual reality (VR) technology, panoramic videos are increasingly used in various fields, such as online shopping, live broadcasts of concerts and competitions, VR games, VR simulations, etc. Each frame in a panoramic video is a rendered panoramic image, which can enhance the user's viewing experience and give the user an immersive feeling.

[0025] At present, the panoramic video production process mainly includes: setting multiple acquisition channels to shoot the target area, each acquisition channel can capture images of different parts or angles of the target area, and then stitching the images captured by multiple acquisition channels to obtain a panoramic video frame.

[0026] The stitching process of panoramic video frames usually merges multiple images and renders the merged image. The rendering process can use the equidistant cylindrical projection method. The equidistant cylindrical projection method is a map projection method. In this projection method, by assuming that the sphere and the cylinder are tangent to the equator, the longitude and latitude lines on the sphere are projected onto the cylindrical surface, and then unfolded into a plane along a mother line of the cylindrical surface. However, when using the equidistant cylindrical projection method, the farther the image is from the equator, the more serious the distortion is during the rendering process. The distorted image cannot be effectively recognized, which affects the accuracy of image recognition.

[0027] In addition, the rendering process can also use a cube projection method, that is, projecting the panoramic image onto the six surfaces of a cube to obtain six images. In this projection method: although each individual image does not have a distortion problem, the calculation is complex and the amount of calculation is large when performing image recognition, and image recognition cannot be performed when the target object appears at the junction of two images.

[0028] Therefore, panoramic videos have problems such as distortion and complex calculations in image recognition due to different rendering methods, making it difficult to perform further image processing and application.

[0029] In order to improve the above-mentioned problems, the inventors proposed the image recognition method, device, electronic device and storage medium provided in the present application, by obtaining the images to be processed of the group to be detected, wherein the group to be detected includes the images to be processed respectively acquired by multiple acquisition channels at the same acquisition time, and then performing image recognition processing on the images to be processed of the group to be detected to obtain the target object recognition result, and performing image stitching processing on the images to be processed of the group to be detected to obtain panoramic video frames, and then associating the panoramic video frames corresponding to the group to be detected with the target object recognition results. By performing image recognition processing on the acquired images to be processed and then associating the image recognition results with the corresponding panoramic video frames, the image recognition difficulty of the panoramic video frames can be simplified and the accuracy of image recognition can be improved.

[0030] The image recognition method provided in the embodiment of the present application is applicable to any scenario of generating panoramic video. For example, it can be applied to the field of panoramic video Internet live broadcast technology.

[0031] In order to facilitate the explanation of the technical solution of the present application, one application environment of the image recognition method provided by the present invention is first introduced below with reference to the accompanying drawings.

[0032] See also Figure 1 , Figure 1 The structure diagram of a live broadcast network system 100 provided in an embodiment of the present application. The live broadcast network system 100 may include an anchor terminal 110, a server 120 and an audience terminal 130. It should be noted that: Figure 1 This is an exemplary application environment. The method provided in the embodiment of the present application can also be run in other application environments, and the present application does not limit this.

[0033] The server 110 may be a single server (network access server), a server cluster consisting of several servers (cloud server), or a cloud computing center (database server).

[0034] The host terminal 120 and the viewing terminal 130 may be any device with communication and storage functions, including but not limited to a PC (Personal Computer), a PDA (Tablet Computer), a smart TV, a smart phone, a smart wearable device, or other smart communication devices with network connection functions. In some embodiments, the number of the host terminal 110 is one or more. The number of the viewing terminal 130 may also be one or more.

[0035] In the embodiment of the present application, the host terminal 110 and the audience terminal 130 may install a live broadcast application, and the server 120 may provide a live broadcast service to the live broadcast application. For example, the host user may use the host live broadcast function provided by the live broadcast application to broadcast a panoramic video live on the host terminal 110. For another example, the audience user may use the live broadcast viewing function provided by the live broadcast application to watch a panoramic live video on the audience terminal 130.

[0036] In an embodiment of the present application, the anchor terminal 110 can obtain an image of the target area. In some embodiments, the anchor terminal 110 can be provided with an image acquisition device, and the image of the target area is acquired through the image acquisition device. In other embodiments, the anchor terminal 110 may not be provided with an image acquisition device, but is connected to an external image acquisition device to obtain the image of the target area acquired by the image acquisition device. The image acquisition device can be, for example, a camera, a panoramic camera, etc., and the present application does not limit this. In addition, the number of image acquisition devices can be one or more, and the specific number can be set according to actual use needs.

[0037] In some embodiments, the anchor terminal 110 can obtain the image captured by the image acquisition device through an acquisition card. The acquisition card can be, for example, a USB (Universal Serial BUS) acquisition card, a digital acquisition card, an analog acquisition card, etc. In other embodiments, the anchor terminal 110 can also obtain the image captured by the image acquisition device through a network.

[0038] The anchor terminal 110 obtains images of the target area, stitches the images together to form a panoramic video frame, and then sends the panoramic video frame to the server 120. The server 120 sends the received panoramic video frame to the viewer terminal 130, and the viewer user can watch the content of the panoramic video frame on the viewer terminal 130.

[0039] Among them, the anchor terminal 110 and the server 120, and the audience terminal 130 and the server 120 can communicate with each other through the network. Optionally, the above-mentioned Internet uses standard communication technology and / or protocol. The Internet is usually the Internet, but it can also be any network, including but not limited to any combination of local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired or wireless network, private network or virtual private network. In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies may be used to replace or supplement the above-mentioned data communication technologies.

[0040] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0041] See also Figure 2 An image recognition method provided in an embodiment of the present application can be applied to the above-mentioned live broadcast network system. In the method provided in an embodiment of the present application, the execution subject of the method is an end that generates a panoramic video frame, for example, it can be the above-mentioned anchor terminal.

[0042] When watching a panoramic video, the audience can select a viewing angle and area on the panoramic video. Different audiences can select different viewing angles and areas, that is, different audiences present different viewing angles and areas of the panoramic video on their respective audience terminals. If it is at one end of the panoramic video frame being played, such as the above-mentioned audience terminal, image recognition of the panoramic video frame is performed. On the one hand, the viewing angle of each audience terminal is different, and the image recognition results obtained from different viewing angles are different. A large amount of calculations are required for image recognition for each audience terminal. On the other hand, the types and performances of the audience terminals are different. Performing image recognition of panoramic video frames on low-configuration audience terminals will cause the audience terminal resources to be occupied too much, resulting in problems such as stuttering, and the image recognition results cannot be guaranteed, affecting the user experience. In order to solve the above-mentioned technical difficulties, the execution subject of the image recognition method of the embodiment of the present application is one end that generates a panoramic video frame, such as the above-mentioned anchor terminal. Thus, the image recognition processing of the panoramic video frame only needs to be uniformly performed on the anchor terminal, and no repeated processing is required for each audience terminal, and the area and viewing angle of the image for image recognition by the anchor terminal are fixed and comprehensive, that is, the image recognition results are unified and stable, which can improve the user experience.

[0043] Specifically, the image recognition method of the embodiment of the present application includes: steps 210 to 240.

[0044] In step 210, the images to be processed of the group to be detected are obtained, wherein the group to be detected includes the images to be processed respectively acquired by multiple acquisition channels at the same acquisition time.

[0045] Different acquisition channels correspond to different acquisition areas or angles, and the acquisition areas or angles between different acquisition channels may partially overlap or may be completely different.

[0046] In some embodiments, the anchor terminal can obtain the image to be processed through an image acquisition device. The number of image acquisition devices can be one or more. Each image acquisition device can include one or more acquisition channels, so as to acquire images of one or more acquisition areas.

[0047] In order to obtain a panoramic video frame, the image acquisition device collects images at a fixed time. At each acquisition moment, the image acquisition device collects a group of images, and each group of images includes an image collected by each acquisition channel at the current acquisition moment. In an embodiment of the present application, the group to be detected includes images to be processed that are collected by multiple acquisition channels at the same acquisition moment.

[0048] Furthermore, the acquired images may be stored in a storage space of the anchor terminal, such as a memory, a buffer area, etc. When processing the images, the images to be processed of the group to be detected may be acquired from the buffer area for processing.

[0049] In step 220, image recognition processing is performed on the to-be-processed images of the to-be-detected group to obtain target object recognition results.

[0050] In the embodiment of the present application, the images to be processed of the group to be detected are the original images acquired by the image acquisition device, and have not yet been spliced. Therefore, the images to be processed of the group to be detected will not have problems such as distortion caused by the splicing process. Performing image recognition on the images to be processed of the group to be detected can simplify the difficulty of image recognition and improve the accuracy of image recognition.

[0051] Among them, image recognition processing is performed on the to-be-processed images of the to-be-detected group to obtain target object recognition results to determine whether there is a target object in the to-be-processed images. When the target object exists in the to-be-processed images, the position of the target object in the to-be-processed images can be further determined. Furthermore, according to the position of the to-be-processed images in the panoramic video frame, the specific position of the target object in the panoramic video frame can be determined.

[0052] The target object may include a person, a specific object, etc., and may be set according to actual use needs. During live broadcasting, the target object may be, for example, the face of the anchor. The number of target objects may be one or more. For example, when the target object is the anchor, the anchor may be one person or multiple people.

[0053] Specifically, image recognition processing can be implemented using template matching algorithm, Eigenface algorithm (feature face method), Fisherfaces algorithm, LBPH algorithm (Local Binary Patterns Histograms), or through pre-trained neural network for image recognition processing, which is not limited in this application.

[0054] Each group to be detected includes multiple images to be processed. In order to improve the efficiency of image recognition processing, multiple threads can be set to perform image recognition processing on multiple images to be processed in the group to be detected. That is, image recognition processing is performed on multiple images to be processed at the same time each time. Among them, the number of threads can be set according to actual use needs, and this application does not limit this.

[0055] In some embodiments, the number of target objects may be one. When performing image recognition processing on multiple images to be processed in the detection group, if the target object is recognized, the image recognition processing of the unprocessed images to be processed in the detection group may be stopped, thereby improving efficiency.

[0056] In some embodiments, the number of target objects may be multiple. If the specific number of target objects is known, then when performing image recognition processing on multiple images to be processed of the detection group, if a corresponding number of target objects are identified, the image recognition processing of the unprocessed images to be processed of the detection group can be stopped. If the specific number of target objects is not determined, then when performing image recognition processing on multiple images to be processed of the detection group, image recognition processing needs to be performed on each image to be processed in the detection group.

[0057] Multiple images to be processed cover different areas and different angles of the target area. However, in some cases, there is no target object in some areas, such as invalid areas such as the ceiling area and the inactive area of ​​the target object. It is impossible for the target object to exist in these invalid areas. It is impossible to identify the target object by performing image recognition processing on the images corresponding to the invalid areas, which takes up computing resources and reduces the efficiency of target object recognition. In order to solve this technical problem, in some embodiments, the invalid area can be determined in advance. For example, the user can set it in advance in the interactive interface that displays the acquisition area information and select part of the acquisition area as the invalid area. Furthermore, when performing image recognition processing, only the areas outside the invalid areas in the image to be processed are processed, thereby reducing the amount of computation for image recognition processing and improving the efficiency of image recognition.

[0058] When performing image recognition processing, the multiple images to be processed in the detection group can be processed in a fixed order, or in a random order.

[0059] In order to improve the efficiency of image recognition processing, image recognition processing may also be performed preferentially on images to be processed in which the probability of the target object appearing is high.

[0060] As an implementation manner, image recognition processing is performed on the to-be-processed images of the to-be-detected group to obtain target object recognition results, including: steps 2211 to 2213.

[0061] In step 2211, the images to be processed in the group to be detected are sorted to obtain a sorting queue.

[0062] In step 2212, image recognition processing is performed on the images to be processed in the sorting queue in sequence.

[0063] In step 2213, if the target object is identified, the location information of the target object is determined.

[0064] The images to be processed in the group to be detected can be sorted to obtain a sorting queue. The sorting of the images to be processed in the group to be detected can be sorted in a fixed order, a random order, or according to the probability of the target object appearing.

[0065] In some embodiments, sorting the images to be processed of the detection group to obtain a sorting queue may be to sort the images to be processed according to the target score of the acquisition channel. The target score of the acquisition channel may represent the probability of the target object appearing in the area acquired by the acquisition channel. The higher the target score of the acquisition channel, the higher the probability of the target object appearing, and the higher the ranking of the images to be processed acquired by the acquisition channel. Sorting by the target score of the acquisition channel and giving priority to the images to be processed with a higher probability of the target object appearing can increase the rate of identifying the target object.

[0066] As an example, when the target object is not recognized, the initial values ​​of the target scores of the acquisition channels may be the same. In this case, the multiple images to be processed in the detection group may be sorted in a random order.

[0067] As an example, the user may also divide the target area in advance, for example, the target area may be divided into an active area where the target object is active, and an invalid area where the target object has a lower probability of appearing. The larger the range of the active area included in the acquisition channel, the higher the corresponding target score. The larger the range of the invalid area included in the acquisition channel, the lower the corresponding target score.

[0068] As an example, if a target object is identified, the processing order of the subsequently acquired to-be-processed images of the to-be-detected group may be adjusted with reference to the location of the identified target object.

[0069] It can be understood that if the position of the target object does not change, the probability of identifying the target object at the same position in the subsequently acquired to-be-processed image of the to-be-detected group is relatively high.

[0070] If the position of the target object changes, the target object also moves to the surrounding area from the original position as the starting point, then the probability of identifying the target object in the area near the corresponding position in the subsequently acquired image to be processed is relatively high. Therefore, when the target object is identified, the position information of the target object can be determined, and the target score of each subsequent acquisition channel can be adjusted according to the position of the target object. The following is an explanation with a specific example.

[0071] As an implementation method, after sequentially performing target object recognition processing on the images to be processed in the sorting queue, the image recognition method further includes: if the target object is recognized, adjusting the target scores of each acquisition channel. If the target object is recognized, after determining the position information of the target object, the image recognition method further includes: determining the detection group that has not been subjected to image recognition processing as the group to be detected, and returning to the step of obtaining the images to be processed of the group to be detected, that is, adjusting the processing order of subsequent images to be processed according to the adjusted target scores of each acquisition channel.

[0072] Among them, when the target object is identified, the target score of each acquisition channel can be adjusted in time according to the location of the target object, so that the target object can be quickly locked when the image recognition processing is performed on the subsequent group to be detected.

[0073] Specifically, one or more influencing factors affecting the target score of the acquisition channel can be set.

[0074] As an example, the influencing factor may be, for example, whether the corresponding acquisition channel has acquired the target object. Specifically, if the target object is identified, the target score of the acquisition channel corresponding to the image to be processed with the target object can be adjusted to be higher than the target score of the acquisition channel corresponding to the image to be processed without the target object. Thus, when performing image processing on the subsequent group to be detected, the image to be processed corresponding to the acquisition channel that has acquired the target object is preferentially processed.

[0075] As an example, the influencing factor may also be the area size of the target object collected. Specifically, if it is identified that there are multiple images to be processed with the target object, the target scores of the acquisition channels corresponding to the multiple images to be processed with the target object can be adjusted according to the area of ​​the target object. The larger the area of ​​the target object, the closer the target object is to the image acquisition position, and the more accurate the position of the target object is obtained. Therefore, the larger the area of ​​the target object, the higher the target score of the acquisition channel corresponding to the image to be processed where the target object is located.

[0076] As an example, the influencing factor may also be the movement trend of the target object. Specifically, when the target object is identified, the moving direction of the target object may be predicted, and the moving area of ​​the target object may be predicted based on the moving direction. Then, if the collection area of ​​the collection channel includes the larger area of ​​the moving area, the higher the target score corresponding to the collection channel.

[0077] Optionally, when the target object is identified, key point information of the target object can be obtained. For example, when the target object is a face, key point information such as eyes, nose, and mouth can be obtained, and the orientation range of the face can be determined based on the key point information. The area within the orientation range of the face is the predicted moving area of ​​the target object. Specifically, the more angles that fall into the orientation range of the face, the higher the target score of the acquisition channel corresponding to the acquisition area. Figure 4 As shown, a face image is recognized in the acquisition area B1, and the direction range C of the face is determined. Figure 4 In the figure, the angle of the acquisition area B2 falling into the face orientation range C is greater than the angle of the acquisition area B3 falling into the face orientation range C, so the target score of the acquisition channel corresponding to the acquisition range B2 is higher than the target score of the acquisition channel corresponding to the acquisition range B3.

[0078] Optionally, when the target object is recognized in all adjacent images to be detected, if the position of the target object changes, the moving direction of the target object can be determined in combination with the position change of the target object, and the acquisition area pointed to by the moving direction of the target object is the predicted moving area of ​​the target object. Figure 5 As shown, the acquisition area pointed to by the moving direction D of the target object is B2, and the acquisition area B1 is the predicted moving area of ​​the target object, so the target score of the acquisition channel corresponding to the acquisition range B2 is higher than the target scores of other acquisition channels.

[0079] As an example, the influencing factor may also be the positional relationship with the area where the target object is located. It is understandable that if the position of the target object changes, it moves from the original position to the surrounding area. Therefore, the closer to the area where the target object is located, the higher the probability of the target object appearing. Specifically, when the target object is identified, the closer the acquisition area of ​​the acquisition channel is to the area where the target object is located, the higher the target score corresponding to the acquisition channel.

[0080] It is understandable that in other implementations, other influencing factors may be used to adjust the target score of the acquisition channel, such as the movement posture of the foot, voice keywords spoken by the target object, etc., and this application does not limit this.

[0081] In some embodiments, if multiple influencing factors are used, a weight value can also be set for each influencing factor. Different weight values ​​can be set according to the importance of the influencing factors. For example, the target score is S, and the influencing factors include the distance factor from the area where the target object is located and the trend factor of the movement trend of the target object. Set the weight of the distance factor to w1 and the weight of the trend factor to w2, then the target score S = w1*S1+w2*S2, S1 is the score on the distance factor, and S2 is the score on the trend factor.

[0082] The following is an example. When the acquisition channel corresponding to the image to be processed acquires the target object, assuming the full score is 100, the target score of the acquisition channel is S = 100. Then, the target scores of other acquisition channels where there is no target object can be determined based on the distance factor and the trend factor. Assume that the full score of the distance factor is 100, and the weight value of the distance factor is 0.6. The full score of the trend factor is 100, and the weight value of the trend factor is 0.4.

[0083] In the remaining acquisition channels, the closer the acquisition area is to the target object area, the higher the score on the distance factor. The closest to the target object area is 100. The more the acquisition area falls into the predicted moving area of ​​the target object, the higher the score on the trend factor. The score completely falls into the predicted moving area of ​​the target object is 100. Assuming that acquisition channel F1 acquires the target object, the target score S of acquisition channel F1 is F1 = 100. If the target object is not captured by acquisition channel F2, but the score on the distance factor is 80 and the score on the trend factor is 90, then the target score S of acquisition channel F2 is F2 =80*0.6+90*0.4=84.

[0084] By adopting multiple types of influencing factors and setting the weights of the influencing factors, the probability of the target object appearing in each acquisition area can be more effectively calculated, and then the order of processing the images to be processed can be determined according to the probability of the target object appearing. This can quickly identify the target object and improve the efficiency of image processing.

[0085] Usually, the image recognition processing can be performed on the detection group in the order of the acquisition time. As an implementation method, all the images to be processed in the detection group can be sequentially subjected to image recognition processing in the order of the acquisition time. As another implementation method, the detection group can also be subjected to frame skipping processing, that is, part of the detection group can be subjected to image recognition processing without image recognition processing, and the subsequent detection group can be subjected to image recognition processing directly. For example, for the detection groups a1, a2 and a3, after the detection group a1 is subjected to image recognition processing, the detection group a2 can be directly skipped, and the detection group a3 can be subjected to image recognition processing. As another implementation method, image recognition processing can also be performed on only part of the images to be processed in the detection group each time, and polling recognition is performed in sequence until the target object is identified. For example, in the above-mentioned detection groups a1 and a2, each detection group includes 6 images to be processed. Image recognition processing can be performed on 3 of the images to be processed in the detection group a1 first. Then image recognition is performed on 3 of the images to be processed in the detection group a2. Then, the process returns to perform image recognition processing on the remaining three images to be processed in the detection group a1, and performs image recognition processing on the remaining three images to be processed in the detection group a2.

[0086] When performing image recognition processing on the group to be detected, since the acquisition areas corresponding to some acquisition channels may overlap, it is possible that the images to be processed include repeated images. When the target object appears in the above-mentioned overlapping area, the same target object may be identified in multiple images to be processed. At this time, one of the images to be processed that recognizes the target object can be selected, and the image recognition result obtained by performing image recognition processing on the image to be processed is used as the target object recognition result. The following is an explanation with a specific example.

[0087] As an implementation manner, image recognition processing is performed on the to-be-processed images of the to-be-detected group to obtain target object recognition results, including: steps 2221 to 2223.

[0088] Step 2221: If a target object is identified in a plurality of images to be processed in the group to be detected, the position of each target object in the preset coordinate system is determined respectively.

[0089] Step 2222, determine the distance between the locations.

[0090] Step 2223: If the distance is less than the preset distance threshold, determine the location information of the target object with the largest area.

[0091] Wherein, when a target object is identified in a plurality of images to be processed, the position of each target object in a preset coordinate system can be determined respectively. That is, the position of the target object identified in the plurality of images to be processed in the same coordinate system is determined. If the distance between the positions of the target objects is less than a preset distance threshold, the target objects identified in the plurality of images to be processed are determined to be the same target object. Optionally, other factors can also be combined to determine whether they are the same target object, for example, the area of ​​the target object identified in the plurality of images to be processed is determined. For example, when the target object is a face, the area of ​​the circumscribed rectangle of the face can be identified. If the distance between the positions of the target objects is less than a preset distance threshold, and the difference in the area between the positions of the target objects is less than a preset threshold, the target objects identified in the plurality of images to be processed are determined to be the same target object. The preset coordinate system can be, for example, a world coordinate system, and the coordinates of the target object in the world coordinate system can be used to represent the position of the target object. Optionally, the distance between the positions of the target objects can be determined according to the Euclidean distance calculation formula. The larger the target object, the closer it is to the image acquisition device, and the clearer the target object captured by the image acquisition device. Therefore, when the distance is less than the preset distance threshold, the position information of the target object with the largest area can be used as the target object recognition result.

[0092] When there are a large number of images to be processed or a large number of image recognitions are required, multiple images to be processed may be combined and image recognition may be performed on the combined images to reduce the number of image recognitions and improve the efficiency of image recognition. A specific example is given below to illustrate this.

[0093] As an implementation manner, performing image recognition processing on the to-be-processed images of the to-be-detected group to obtain target object recognition results includes: steps 2231 to 2233.

[0094] Step 2231: Group the images to be processed in the group to be detected, wherein each group of images to be processed includes at least two images to be processed.

[0095] Step 2232: Combine each group of images to be processed to form a combined image.

[0096] Step 2233: perform image recognition processing on the combined image to obtain a target object recognition result.

[0097] Among them, different image recognition methods have different restrictions on the images that can be processed. For example, some image recognition algorithms have restrictions on the maximum resolution of the images processed. In this case, the resolution of the combined image formed by each group of images to be processed must be smaller than the maximum resolution corresponding to the image recognition algorithm.

[0098] As an implementation method, when each group of images to be processed is combined to form a combined image, the positional relationship between the combined image and the image to be processed can be recorded. Thus, when a target object is identified in the combined image, the position of the target object in the image to be processed can be determined based on the positional relationship of the target object in the combined image and the positional relationship between the combined image and the image to be processed, and then the position of the target object in the panoramic video frame can be determined.

[0099] The image recognition processing of the combined image can also refer to the method of performing image recognition processing on a single image to be processed, such as setting multiple threads to perform image recognition processing on multiple combined images at the same time, sorting the combined images, etc. Optionally, when performing the sorting processing, the target score of the combined image can be the sum of the target scores of all the images to be processed contained therein.

[0100] In step 230, image stitching processing is performed on the to-be-processed images of the to-be-detected group to obtain a panoramic video frame.

[0101] Among them, the panoramic video frame of the target area can be obtained by stitching and rendering the images to be processed of the detection group. The specific rendering processing method can refer to the above description and will not be repeated here.

[0102] In step 240, the panoramic video frames corresponding to the group to be detected are associated with the target object recognition result.

[0103] The image recognition processing and image stitching processing of the to-be-processed images of the to-be-detected group are performed separately. Therefore, it is necessary to associate the panoramic video frames and target object recognition results corresponding to the same to-be-detected group.

[0104] As an implementation method, the same identifier can be set for the panoramic video frames and target object recognition results corresponding to the same group to be detected to associate the panoramic video frames and target object recognition results corresponding to the same group to be detected. When obtaining the target object recognition results corresponding to the panoramic video frames, the target object recognition results with the same identifier as the panoramic video frames can be selected according to the identifier.

[0105] In some implementations, after step 240, the image recognition method of the embodiment of the present application may further include:

[0106] The target object in the panoramic video frame is subjected to image post-processing according to the target object recognition result associated with the panoramic video frame to obtain a processed panoramic video frame.

[0107] Among them, image post-processing may include but is not limited to beauty processing, filter processing, watermark processing, blur processing, adding target frames, etc. After obtaining the target object recognition result, the target object can be processed accordingly according to actual use needs, thereby enriching the display method of the panoramic video and improving the user's viewing experience.

[0108] In some implementations, it is also possible to interact with the user based on the target object recognition result. For example, Figure 8 In the live broadcast scenario shown, users can interact with the anchor, for example, users can give virtual gifts to their favorite anchor. Figure 8 In the example, the user touches the gift giving control E1, and a gift giving interactive interface E2 appears. The user can choose a gift to reward the anchor, such as a hat, etc., on the gift giving interactive interface E2. Fig. 9 As shown, when the user touches the gift control E3, the position of the anchor can be determined in the subsequent panoramic video frame according to the image recognition result, and a "hat" E4 pattern can be added at the anchor's head position. This enhances the interaction between the anchor and the user, increases the fun of the panoramic video live broadcast process, and improves the user's sense of participation.

[0109] The image recognition method provided by the embodiment of the present application obtains images to be processed of a group to be detected, wherein the group to be detected includes images to be processed that are respectively acquired by multiple acquisition channels at the same acquisition time, and then performs image recognition processing on the images to be processed of the group to be detected to obtain target object recognition results, and performs image stitching processing on the images to be processed of the group to be detected to obtain panoramic video frames, and then associates the panoramic video frames corresponding to the group to be detected with the target object recognition results. By performing image recognition processing on the acquired images to be processed and then associating the image recognition results with the corresponding panoramic video frames, the image recognition difficulty of the panoramic video frames can be simplified and the accuracy of image recognition can be improved.

[0110] See also Fig.10 , which shows an image recognition device provided by an embodiment of the present invention. The image recognition device 300 includes: an image acquisition module 310, an image recognition module 320, an image stitching module 330 and an association module 340.

[0111] The image acquisition module 310 is used to acquire the images to be processed of the group to be detected, wherein the group to be detected includes the images to be processed acquired by multiple acquisition channels at the same acquisition time.

[0112] The image recognition module 320 is used to perform image recognition processing on the to-be-processed images of the to-be-detected group to obtain target object recognition results.

[0113] The image stitching module 330 is used to perform image stitching processing on the to-be-processed images of the to-be-detected group to obtain a panoramic video frame.

[0114] The association module 340 is used to associate the panoramic video frame corresponding to the group to be detected with the target object recognition result.

[0115] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Any processing method described in the method embodiment can be implemented by the corresponding processing module in the device embodiment, and will not be repeated one by one in the device embodiment.

[0116] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0117] See also Fig.11 Based on the above image recognition method, the embodiment of the present application also provides another electronic device 400 including a processor that can execute the above image recognition method, and the electronic device 400 also includes one or more processors 410, a memory 420, and one or more application programs. The memory 420 stores a program that can execute the content of the above embodiment, and the processor 410 can execute the program stored in the memory.

[0118] Among them, the processor 410 may include one or more cores for processing data and a message matrix unit. The processor 410 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions and processes data of the electronic device 400 by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor 410 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but may be implemented separately through a communication chip.

[0119] The memory 420 may include a random access memory (RAM) or a read-only memory (ROM). The memory 420 may be used to store instructions, programs, codes, code sets or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as an image acquisition function, an image recognition function, an image stitching function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the terminal during use (such as images to be processed, panoramic video frames, location information, target scores, etc.).

[0120] Please refer to Fig.12, which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium stores program codes, which can be called by a processor to execute the image recognition method described in the above method embodiment.

[0121] The computer readable storage medium may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium includes a non-transitory computer-readable storage medium. The computer readable storage medium has storage space for program codes that execute any of the method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.

[0122] In summary, the present application provides an image recognition method, device, electronic device and storage medium, which obtains the images to be processed of the group to be detected, wherein the group to be detected includes the images to be processed respectively acquired by multiple acquisition channels at the same acquisition time, and then performs image recognition processing on the images to be processed of the group to be detected to obtain the target object recognition result, and performs image stitching processing on the images to be processed of the group to be detected to obtain panoramic video frames, and then associates the panoramic video frames corresponding to the group to be detected with the target object recognition results. By performing image recognition processing on the acquired images to be processed and then associating the image recognition results with the corresponding panoramic video frames, the image recognition difficulty of the panoramic video frames can be simplified and the accuracy of image recognition can be improved.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image recognition method, characterized in that: The image recognition method comprises: Acquire the images to be processed of the group to be detected; wherein the group to be detected includes the images to be processed respectively acquired by multiple acquisition channels at the same acquisition time; Performing image recognition processing on the to-be-processed images of the to-be-detected group to obtain target object recognition results; performing image recognition processing on the to-be-processed images of the to-be-detected group to obtain target object recognition results, including: determining whether there is a target object in the to-be-processed image; when determining that there is a target object in the to-be-processed image, determining the position of the target object in the to-be-processed image; and determining the position of the target object in the panoramic video frame according to the position of the target object in the to-be-processed image; Performing image stitching processing on the to-be-processed images of the to-be-detected group to obtain the panoramic video frame; wherein the image recognition processing and the image stitching processing are performed separately; Associating the panoramic video frame corresponding to the to-be-detected group with the target object recognition result; Image post-processing is performed on the target object in the panoramic video frame according to the target object recognition result associated with the panoramic video frame to obtain a processed panoramic video frame.

2. The image recognition method according to claim 1, characterized in that: The performing image recognition processing on the to-be-processed images of the to-be-detected group to obtain target object recognition results includes: Sorting the to-be-processed images of the to-be-detected group to obtain a sorting queue; Performing image recognition processing on the images to be processed in the sorting queue in sequence; If the target object is identified, the location information of the target object is determined.

3. The image recognition method according to claim 2, characterized in that: The step of sorting the to-be-processed images of the to-be-detected group to obtain a sorted queue comprises: The images to be processed are sorted according to the target scores of the acquisition channels, wherein the higher the target score of the acquisition channel is, the higher the ranking of the images to be processed acquired by the acquisition channel is.

4. The image recognition method according to claim 2, characterized in that: After sequentially performing target object recognition processing on the to-be-processed images in the sorting queue, the image recognition method further includes: If the target object is identified, adjusting the target score of each acquisition channel; After determining the location information of the target object if the target object is identified, the method further includes: The detection group that has not been subjected to the image recognition processing is determined as the group to be detected, and the step of obtaining the image to be processed of the group to be detected is returned to be executed.

5. The image recognition method according to claim 4, characterized in that: If the target object is identified, the target score of each acquisition channel is adjusted, including: If the target object is identified, the target score of the acquisition channel corresponding to the image to be processed with the target object is adjusted to be higher than the target score of the acquisition channel corresponding to the image to be processed without the target object.

6. The image recognition method according to claim 5, characterized in that: The image recognition method further comprises: If it is identified that there are multiple images to be processed containing the target object, the target scores of the acquisition channels corresponding to the multiple images to be processed containing the target object are adjusted according to the area of ​​the target object, wherein the larger the area of ​​the target object is, the higher the target score of the acquisition channel corresponding to the image to be processed containing the target object is.

7. The image recognition method according to claim 5, characterized in that: If the target object is identified, adjusting the target score of each acquisition channel also includes: Predicting a moving direction of the target object; Predicting a moving area of ​​the target object according to the moving direction; If the area of ​​the collection region of the collection channel including the moving region is larger, the target score corresponding to the collection channel is higher.

8. The image recognition method according to claim 5, characterized in that: If the target object is identified, adjusting the target score of each acquisition channel also includes: If the acquisition area of ​​the acquisition channel is closer to the area where the target object is located, the target score corresponding to the acquisition channel is higher.

9. The image recognition method according to claim 1, characterized in that: The performing image recognition processing on the to-be-processed images of the to-be-detected group to obtain target object recognition results includes: Grouping the images to be processed in the group to be detected; wherein each group of images to be processed includes at least two images to be processed; Combining each group of images to be processed to form a combined image; Perform image recognition processing on the combined image to obtain a target object recognition result.

10. The image recognition method according to claim 1, characterized in that: The performing target object recognition processing on the to-be-processed images of the to-be-detected group to obtain a target object recognition result includes: Image recognition processing is performed on the area other than the invalid area in the to-be-processed image of the to-be-detected group to obtain a target object recognition result.

11. The image recognition method according to claim 1, characterized in that: The performing image recognition processing on the to-be-processed images of the to-be-detected group to obtain target object recognition results includes: If the target object is identified in the plurality of images to be processed in the group to be detected, respectively determining the position of each target object in a preset coordinate system; determining a distance between said locations; If the distance is less than a preset distance threshold, the position information of the target object with the largest area is determined.

12. An image recognition device, characterized in that: The image recognition device comprises: An image acquisition module, used to acquire the images to be processed of the group to be detected; wherein the group to be detected includes the images to be processed respectively acquired by multiple acquisition channels at the same acquisition time; An image recognition module is used to perform image recognition processing on the to-be-processed images of the to-be-detected group to obtain a target object recognition result; the image recognition processing on the to-be-processed images of the to-be-detected group to obtain a target object recognition result includes: determining whether there is a target object in the to-be-processed image; when it is determined that there is a target object in the to-be-processed image, determining the position of the target object in the to-be-processed image; and determining the position of the target object in the panoramic video frame according to the position of the target object in the to-be-processed image; An image stitching module, used for performing image stitching processing on the to-be-processed images of the to-be-detected group to obtain the panoramic video frame; wherein the image recognition processing and the image stitching processing are performed separately; An associating module, used to associate the panoramic video frame corresponding to the to-be-detected group with the target object recognition result; The image recognition device is further used to perform image post-processing on the target object in the panoramic video frame according to the target object recognition result associated with the panoramic video frame to obtain a processed panoramic video frame.

13. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more applications are configured to execute the image recognition method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the image recognition method according to any one of claims 1 to 11.

Images